U.S. TIMSS and PIRLS 2011 
Technical Report and User’s 
Guide 

September 2013 


David Kastberg 
Stephen Roey 
David Ferraro 
Nita Lemanski 

Westat 

Ebru Erberber 

AIR 

Stephen Provasnik 
Sheila Thompson 

Project Officers 

National Center for Education Statistics 

NCES 2013-046 

U.S. DEPARTMENT OF EDUCATION 





NATIONAL CENTER for 
EDUCATION STATISTICS 


Institute of Education Sciences 


U.S. Department of Education 

Arne Duncan 
Secretary 

Institute of Education Sciences 

John Q. Easton 
Director 

National Center for Education Statistics 

Jack Buckley 
Commissioner 

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and 
reporting data related to education in the United States and other nations. It fulfills a congressional mandate to 
collect, collate, analyze, and report full and complete statistics on the condition of education in the United States; 
conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state 
and local education agencies in improving their statistical systems; and review and report on education activities in 
foreign countries. 

NCES activities are designed to address high-priority education data needs; provide consistent, reliable, complete, 
and accurate indicators of education status and trends; and report timely, useful, and high-quality data to the U.S. 
Department of Education, the Congress, the states, other education policymakers, practitioners, data users, and the 
general public. Unless specifically noted, all information contained herein is in the public domain. 

We strive to make our products available in a variety of formats and in language that is appropriate to a variety of 
audiences. You, as our customer, are the best judge of our success in communicating information effectively. If you 
have any comments or suggestions about this or any other NCES product or report, we would like to hear from you. 
Please direct your comments to: 

NCES, lES, U.S. Department of Education 
1990 K Street NW 
Washington, DC 20006-5651 

September 2013 

The NCES Home Page address is http://nces.ed.gov . 

The NCES Publications and Products address is http://nces.ed.gov/pubsearch . 

This publication is only available online. To download, view, and print the report as a PDF file, go to the NCES 
Publications and Products address shown above. 

This report was prepared in part under Contract No. ED-04-CO-0059/0026 with Westat. Mention of trade names, 
commercial products, or organizations does not imply endorsement by the U.S. Government. 

Suggested Citation 

Kastberg, D., Roey, S., Ferraro, D., Lemanski, N., and Erberber, E. (2013). U.S. TIMSS and PIRLS 2011 Technical 
Report and User’s Guide. (NCES 2013-046). U.S. Department of Education. Washington, DC: National Center for 
Education Statistics. Available from http://nces.ed.gov/pubsearch . 

Content Contact 

Stephen Provasnik 
(202) 502-7480 
Stephen.Provasnik@ed. gov 


Sheila Thompson 
(202) 502-7425 
Sheila. Thompson@ed. gov 


Contents 


Chapter Page 


LIST OF TABLES v 

LIST OF EXHIBITS x 

1 INTRODUCTION 1 

LI TIMSS in Brief I 

1.2 PIRLS in Brief 2 

1.3 TIMSS and PIRLS 2011 U.S. and State Benchmarking Data 

Collection Activities and Schedule 3 

1 .4 Overview of the Design and Administration of TIMSS 2011 4 

1.5 Reporting TIMSS and PIRLS Results 7 

1.6 U.S. International and National Data Files 7 

2 SAMPLING SCHOOLS 9 

2.1 School Sampling Frames 12 

2.2 National School Samples 20 

2.3 State School Samples 30 

3 PARTICIPATION RATES AND NONRESPONSE BIAS 35 

3.1 TIMSS and PIRES Participation Rates of U.S. Schools, 

Classrooms, and Students 36 

3.2 Participation Rates for All Countries 40 

3.3 Exclusions 41 

3.4 Nonresponse Bias Analysis 42 

3.5 Technical Notes 77 

4 SURVEY OPERATIONS 86 

4.1 Recruiting Districts and Schools 86 

4.2 Sampling Students Within Schools 93 

4.3 Instruments for TIMSS and PIRLS 103 

4.4 Field Operations 110 

4.5 Receipt Control, Scoring, Coding, and Data Entry 118 

4.6 Data Preparation 124 

5 THE TIMSS AND PIRES 20 1 1 DATA FOR THE UNITED 

STATES AND BENCHMARKING STATES 130 

5. 1 U.S. International, National, and State Benchmarking Data Files 130 

5.2 TIMSS and PIRES Data Files 132 

5.3 TIMSS and PIRES Achievement Data Files and Variable Names 135 

5.4 TIMSS and PIRES Background Questionnaire Data Files 138 


iii 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 



Contents (continued) 


Chapter Page 


5.5 Variable Naming Convention for Background Variables 140 

5.6 Sampling and Weighting Variables 142 

5.7 Structure and Design Variables in TIMSS and PIRLS 2011 

Data Files 144 

5.8 TIMSS and PIRLS 2011 Codebook Files 145 

5.9 The U.S. National and State Benchmarking Data Files 145 

5.10 Data Files Available From the Website of the National 

Center for Education Statistics 153 

5.1 1 Merging TIMSS/PIRLS 2011 Data Files 154 

5.12 Some Notes on Analyzing the TIMSS and PIRLS 2011 Data 162 

References R-1 


List of Appendixes 

Appendix 

A State Sampling Tables A-1 

B State Participation Rates B-1 

C TIMSS & PIRLS 201 1 Recruitment Materials C-1 

D Informational Materials D-1 

E Field Staff Training Materials E-1 

F U.S. Versions of TIMSS and PIRLS 2011 Student, School, Teacher 

and Curriculum Questionnaires F-1 

G TIMSS and PIRLS 201 1 Questionnaire Adaptations G-1 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


IV 



List of Tables 


Table Page 


2-1 Number and percentage of students and schools included in the U.S. 

TIMSS-PIRLS fourth-grade school sampling frame, by grade span: 

2011 13 

2-2 Number and percentage of students and schools included in the U.S. 

TIMSS eighth-grade school sampling frame, by grade span: 2011 13 

2-3 Number and percentage of students and schools included in the U.S. 

TlMSS-PlRLS fourth-grade school sampling frame, by region: 2011 15 

2-4 Number and percentage of students and schools included in the U.S. 

TIMSS-PIRLS fourth-grade school sampling frame, by poverty 

level: 20 11 15 

2-5 Number and percentage of students and schools included in the U.S. 

TIMSS-PIRLS fourth-grade school sampling frame, by school type: 

2011 15 

2-6 Number and percentage of students and schools included in the U.S. 

TIMSS-PIRLS fourth-grade school sampling frame, by locale: 2011 16 

2-7 Number and percentage of students and schools included in the U.S. 

TIMSS-PIRLS fourth-grade school sampling frame, by race/ethnicity 

status: 20 11 16 

2-8 Number and percentage of students and schools included in the U.S. 

TIMSS-PIRLS fourth-grade school sampling frame, by region, 

poverty level, and school type: 2011 17 

2-9 Number and percentage of students and schools included in the U.S. 

TIMSS eighth-grade school sampling frame, by region: 2011 18 

2-10 Number and percentage of students and schools included in the U.S. 

TIMSS eighth-grade school sampling frame, by poverty level: 2011 18 

2-1 1 Number and percentage of students and schools included in the U.S. 

TIMSS eighth-grade school sampling frame, by school type: 2011 18 

2-12 Number and percentage of students and schools included in the U.S. 

TIMSS eighth-grade school sampling frame, by locale: 2011 19 

2-13 Number and percentage of students and schools included in the U.S. 

TIMSS eighth-grade school sampling frame, by race/ethnicity status: 

2011 19 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


V 



List of Tables (continued) 


Table Page 


2-14 Number and percentage of students and schools included in the U.S. 

TIMSS eighth-grade school sampling frame, by region, poverty 

level, and school type: 2011 20 

2-15 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS-PIRLS 
fourth-grade, by region: 2011 24 

2-16 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TlMSS-PlRLS 
fourth-grade, by poverty level: 2011 24 

2-17 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS-PIRLS 
fourth-grade, by school type: 201 1 25 

2-18 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS-PIRLS 
fourth-grade, by locale: 2011 25 

2-19 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS-PIRLS 
fourth-grade, by race/ethnicity status: 2011 26 

2-20 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS-PIRLS 
fourth-grade, by region, poverty level, and school type: 2011 27 

2-2 1 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS eighth- 

grade, by region: 2011 28 

2-22 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS eighth- 

grade, by poverty level: 2011 28 

2-23 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS eighth- 

grade, by school type: 2011 28 

2-24 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS eighth- 

grade, by locale: 201 1 29 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide vi 



List of Tables (continued) 


Table Page 


2-25 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS eighth- 

grade, by race/ethnicity status: 2011 29 

2-26 Number and percentage of students in the sampling frame and 

number and percentage of schools in the sample, U.S. TIMSS eighth- 

grade, by region, poverty level, and school type: 2011 30 

2- 27 Number of schools included in the state samples, by grade/design, 

state, and sample type: 2011 33 

3- 1 Number of U.S. schools, classrooms, and students participating in 

TIMSS, and participation rates, by grade: 201 1 37 

3-2 Number of U.S. schools, classrooms, and students participating in 

PIRLS, and participation rates, for fourth-grade: 2011 38 

3-3 Percentage distribution of eligible and participating schools in the 

U.S. TIMSS fourth-grade original sample, by selected categorical 
variables: 201 1 47 

3-4 Mean enrollment of eligible and participating schools in the U.S. 

TIMSS fourth-grade original sample: 2011 48 

3-5 Mean percentage of students in eligible and participating schools in 

the U.S. TIMSS fourth-grade original sample, by race/ethnicity: 

2011 49 

3-6 Mean percentage of students eligible for free or reduced-price lunch, 

in eligible and participating public schools in the U.S. TIMSS fourth- 
grade original sample: 2011 49 

3 -7a Logistic regression model parameter estimates (with four 

race/ethnicity variables) using the U.S. TIMSS fourth-grade original 

sample (N = 437): 2011 51 

3 -7b Logistic regression model parameter estimates (with summed 

race/ethnicity percentage) using the U.S. TIMSS fourth-grade 
original sample (N=43 7): 201 1 52 

3-8 Percentage distribution of eligible and participating schools in the 

U.S. TIMSS fourth-grade final sample, by selected categorical 
variables: 201 1 53 


vii 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 



List of Tables (continued) 


Table Page 


3-9 Mean enrollment of eligible and participating schools in the U.S. 

TIMSS fourth-grade final sample: 2011 54 

3-10 Mean percentage of students in eligible and participating schools in 

the U.S. TIMSS fourth-grade final sample, by race/ethnicity: 2011 55 

3-11 Mean percentage of students eligible for free or reduced-price lunch, 

in eligible and participating public schools in the U.S. TIMSS fourth- 
grade final sample: 2011 55 

3- 12a Logistic regression model parameter estimates (with four 

race/ethnicity variables) in the U.S. TIMSS fourth-grade final sample 

(N = 437): 2011 56 

3-1 2b Logistic regression model parameter estimates (with summed 
race/ethnicity percentage) in the U.S. TIMSS fourth-grade final 
sample (N = 437): 2011 57 

3-13 Percentage distribution of eligible and participating schools in the 

U.S. TIMSS fourth-grade nonresponse-adjusted sample, by selected 
categorical variables: 2011 58 

3-14 Mean enrollment of eligible and participating schools in the U.S. 

TIMSS fourth-grade nonresponse-adjusted sample: 201 1 59 

3-15 Mean percentage of students in eligible and participating schools in 

the U.S. TIMSS fourth-grade nonresponse-adjusted sample, by 
race/ethnicity: 2011 60 

3-16 Mean percentage of students eligible for free or reduced-price lunch, 

in eligible and participating public schools in the U.S. TIMSS fourth- 
grade nonresponse-adjusted sample: 2011 60 

3-17 Characteristics with p values less than .05, U.S. TIMSS fourth-grade 

schools: 2011 61 

3-18 Percentage distribution of eligible and participating schools in the 

U.S. PIRLS fourth-grade original sample, by selected categorical 
variables: 201 1 63 

3-19 Mean enrollment of eligible and participating schools in the U.S. 

PIRLS fourth-grade original sample: 2011 64 


viii 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 



List of Tables (continued) 


Table Page 


3-20 Mean percentage of students in eligible and participating schools in 

the U.S. PIRLS fourth-grade original sample, by race/ethnicity: 2011 64 

3-21 Mean percentage of students eligible for free or reduced-price lunch, 

in eligible and participating public schools in the U.S. PIRLS fourth- 
grade original sample: 2011 65 

3 -22a Logistic regression model parameter estimates (with four 

race/ethnicity variables) using the U.S. PIRLS fourth-grade original 

sample (N = 437): 2011 66 

3 -22b Logistic regression model parameter estimates (with summed 

race/ethnicity percentage) using the U.S. PIRLS fourth-grade 
original sample (N = 437): 201 1 67 

3-23 Percentage distribution of eligible and participating schools in the 

U.S. PIRLS fourth-grade final sample, by selected categorical 
variables: 201 1 68 

3-24 Mean enrollment of eligible and participating schools in the U.S. 

PIRLS fourth-grade final sample: 2011 69 

3-25 Mean percentage of students in eligible and participating schools in 

the U.S. PIRLS fourth-grade final sample, by race/ethnicity: 2011 70 

3-26 Mean percentage of students eligible for free or reduced-price lunch, 

in eligible and participating public schools in the U.S. PIRLS fourth- 
grade final sample: 2011 70 

3 -27a Logistic regression model parameter estimates (with four 

race/ethnicity variables) in the U.S. PIRLS fourth-grade final sample 

(N = 437): 2011 71 

3 -27b Logistic regression model parameter estimates (with summed 

race/ethnicity percentage) in the U.S. PIRLS fourth-grade final 
sample (N = 437): 2011 72 

3-28 Percentage distribution of eligible and participating schools in the 

U.S. PIRLS fourth-grade nonresponse-adjusted sample, by selected 
categorical variables: 2011 73 

3-29 Mean enrollment of eligible and participating schools in the U.S. 

PIRLS fourth-grade nonresponse-adjusted sample: 2011 74 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide ix 



List of Tables (continued) 

Table Page 

3-30 Mean percentage of students in eligible and participating schools in 

the U.S. PIRLS fourth-grade nonresponse-adjusted sample, by 
race/ethnicity: 20 1 1 74 

3-3 1 Mean percentage of students eligible for free or reduced-price lunch, 

in eligible and participating public schools in the U.S. PIRLS fourth- 
grade nonresponse-adjusted sample: 2011 75 

3- 32 Characteristics with p values less than .05, U.S. PIRLS fourth-grade 

schools: 2011 75 

List of Exhibits 

Exhibit Page 

1-1 Schedule for U.S. TIMSS and PIRLS 2011 data collection activities 3 

4- 1 Example of TIMSS and PIRLS fourth-grade class listing form with 

mock data 95 

4-2 Example of TIMSS eighth-grade class listing form with mock data 96 

4-3 Example of TIMSS eighth-grade student-teacher linkage form with 

mock data 98 

4-4 Example of PIRLS student listing form with mock data 99 

4-5 Example of TIMSS eighth-grade student tracking form with mock 

data 101 

4- 6 Example of TIMSS eighth-grade teacher tracking form with mock 

data 102 

5- 1 TIMSS data file names 134 

5-2 PIRLS data file names 134 

5-3 Three-digit codes identifying achievement scales in TIMSS and 

PIRLS 137 

5-4 TIMSS and PIRLS sampling weight variables 143 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


X 



List of Exhibits (continued) 

Exhibit Page 

5-5 U.S. national data file names 153 

5-6 Illustrative SAS code for merging U.S. TIMSS eighth-grade student 

and school data 155 

5-7 Illustrative SPSS code for merging U.S. TIMSS eighth-grade student 

and school data 155 

5-8 Illustrative SAS code for merging U.S. TIMSS eighth-grade student 

and mathematics teacher data 156 

5-9 Illustrative SPSS code for merging U.S. TIMSS eighth-grade student 

and mathematics teacher data 157 

5-10 Illustrative SAS code for merging U.S. TIMSS eighth-grade school, 

mathematics teacher and student data 158 

5-1 1 Illustrative SPSS code for merging U.S. TIMSS eighth-grade school, 

mathematics teacher, and student data 159 

5-12 Illustrative SAS code for merging U.S. TIMSS eighth-grade school 

data with restricted-use (CCD and PSS) data 161 

5-13 Illustrative SPSS code for merging U.S. TIMSS eighth-grade school 

data with restricted (CCD and PSS) data 162 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide xi 



1 . 


Introduction 


The U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide provides an overview of the design 
and implementation of the Trends in International Mathematics and Science Study (TIMSS) 2011 and the 
Progress in International Reading Literacy Study (PIRLS) 2011 in the United States and the nine 
participating benchmarking states: Alabama, California, Colorado, Connecticut, Florida, Indiana, 
Massachusetts, Minnesota, and North Carolina, along with information designed to facilitate access to the 
U.S. TIMSS and PIRLS 2011 data. 

1.1 TIMSS in Brief 

The Trends in International Mathematics and Science Study (TIMSS) 2011 is the fifth such study since 
this international comparison of student achievement was first conducted in 1995. Developed and 
implemented at the international level by the International Association for the Evaluation of Educational 
Achievement (lEA), an international organization of national research institutions and governmental 
research agencies, TIMSS is used to measure trends in the mathematics and science knowledge and skills 
of fourth- and eighth-graders. 

TIMSS is designed to align broadly with mathematics and science curricula in the participating countries. 
The results, therefore, suggest the degree to which students have learned mathematics and science 
concepts and skills likely to have been taught in school. TIMSS also collects background information on 
students, teachers, curricula, and schools to allow cross-national comparisons of educational contexts 
related to student achievement. In 201 1, there were 54 countries' and 20 other education systems 
(including all nine benchmarking states) that participated in TIMSS at the fourth- or eighth-grade level, 
or both. 

A detailed treatment of TIMSS 2011 from an international perspective can be found in three reports 
published by the lEA and available online at http://timssandpirls.bc.edu/timss20 1 1/index.html . 

■ TIMSS 2011 International Results in Mathematics (Mullis, Martin, Foy, & Arora, 

2012 ); 


'in this report, “countries” are complete, independent political entities, whereas “other education systems” represent a portion of a country, 
nation, kingdom, or emirate or are other non-national entities (e.g., U.S. states, Canadian provinces, Flemish Belgium, and Northern Ireland). 
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■ TIMSS 2011 International Results in Science (Martin, Mullis, Foy, & Stance, 2012); 
and 

■ Methods and Procedures in TIMSS and PIRLS 2011 (Martin & Mullis, 201 1). 

A U.S. national report is available as well. It is tiX\Qd Highlights From TIMSS 2011: Mathematics and 
Science Achievement of U.S. Fourth- and Eighth-Grade Students in an International Context (Provasnik 
et al., 2012). This report describes the performance of U.S. students relative to their peers in other 
education systems, changes in mathematics and science achievement since 1995, and additional details 
about the achievement of U.S. students nationally and in the benchmarking states that are not available in 
the international reports. The report is available online at 
http://nces.ed.gov/pubsearch/pubsinfo. asp ?pubid=20 13009 . 


1.2 PIRLS in Brief 

The Progress in International Reading Literacy Study (PIRLS) is an international comparative study of 
student achievement. PIRLS 2011 represents the third such study since PIRLS was first conducted in 
200 1 . Developed and implemented by the International Association for the Evaluation of Educational 
Achievement (lEA), an international organization of national research institutions and governmental 
research agencies, PIRLS is used to measure the reading knowledge and skills of fourth-graders over 
time. 

PIRLS is designed to align broadly with reading curricula in the participating education systems. The 
results, therefore, suggest the degree to which students have learned reading concepts and skills likely to 
have been taught in school. PIRLS also collects background information on students, teachers, schools, 
curricula, and official education policies to allow cross-national comparison of educational contexts that 
may be related to student achievement. In 20 1 1 , there were 40 countries and 1 3 other education systems 
(including Florida) that participated in PIRLS. 

A detailed treatment of PIRLS 2011 from an international perspective can be found in two reports 
published by the lEA and available online at http://timssandpirls.bc.edu/pirls20 1 1/index.html . 

■ PIRLS 2011 International Results in Reading (Mullis, Martin, Foy, & Drucker, 2012); 
and 

■ Methods and Procedures in TIMSS and PIRLS 2011 (Martin & Mullis, 201 1). 
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Also available is a U.S. national report, Highlights From PIRLS 2011: Reading Achievement ofU.S. 
Fourth-Grade Students in an International Context (Thompson et al., 2012). This report describes the 
performance of U.S. students relative to their peers in other countries, changes in reading achievement 
since 2001, and additional details about the achievement of U.S. students nationally and in Florida that 
are not available in the international reports. The report is available online at 
http://nces.ed. gov/pubsearch/pubsinfo.asp?pubid=20 13010 . 


1.3 TIMSS and PIRLS 2011 U.S. and State Benchmarking Data 
Collection Activities and Schedule 


Descriptions of data collection activities and their timing within the United States provide a foundation 
for researchers seeking to understand the detail of the data. These activities are listed in exhibit 1-1 along 
with the timing of their implementation. The activities are described in detail in chapter 4 of this report. 


Exhibit 1-1. Schedule for U.S. TIMSS and PIRUS 2011 data collection activities 



SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in International 
Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy Study (PIRLS), 2011. 
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1.4 


Overview of the Design and Administration of TiMSS 2011 


The basic parameters of the design and administration of TIMSS and PIRLS 2011 in the United States are 
outlined below. A more detailed treatment is provided in subsequent chapters of this report. Methods and 
Procedures in TIMSS and PIRLS 2011 (Martin & Mullis, 2011) contains detail on the international 
design. 

Sampling 

Participating countries administered TIMSS and PIRLS to national probability samples of students and 
schools, and other education systems administered TIMSS and PIRLS to comparably representative 
samples. At grade 4 countries had the option of participating in both TIMSS and PIRLS and were given 
options for sampling by the TIMSS and PIRLS International Study Center at Boston College. Several 
countries choose to administer TIMSS and PIRLS in the same schools and assess the same students in 
both studies. The United States selected a grade 4 sample of schools and administered TIMSS and PIRLS 
within the same schools but to different students by selecting classrooms within schools to be assessed in 
either TIMSS or PIRLS. ^ All education systems were required to draw samples of students who were 
nearing the end of their fourth year (TIMSS and PIRLS) or eighth year (TIMSS) of formal schooling. 
Formal schooling was defined as beginning with Level 1 as defined in the International Standard 
Classification of Education — ISCED (UNESCO, 1999). In most education systems, including the United 
States, these students were in the fourth and eighth grades. The U.S. national sample included both public 
and private schools, randomly selected and weighted to be representative of the nation. However, the U.S. 
state samples were representative state samples only of public school students. In total, 369 U.S. schools 
and 12,569 fourth-grade students, along with 501 schools and 10,477 eighth-grade students, participated 
in TIMSS 2011 in the national sample. In PIRLS, 370 schools and 12,726 students participated in the 
national sample. (Counts for schools and students participating in each of the nine benchmarking states 
can be found in Appendix A.) The following chapters present detailed information on sampling 
(chapter 2), participation and response rates (chapter 3), administration (chapter 4), and other technical 
issues. 


^While the majority of the school sample was the same between TIMSS and PIRLS, the composition of schools differed slightly 
due to small schools with only one class that were assigned to one or the other study during class sampling. 
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Assessment Design 


Data for TIMSS and PIRLS 2011 were collected through paper-and-pencil assessments administered to 
the students and through questionnaires completed by principals, teachers, and students. To keep the 
testing burden to a mi nimum and to ensure broad subject-matter coverage, TIMSS used a matrix sampling 
approach where the pool of mathematics and science items at each grade level are organized into a set of 
14 student achievement booklets, with each student completing only one booklet. Each mathematics and 
science item appears in two of the booklets in order to link together the student responses from different 
booklets. The booklets are distributed randomly among students in participating classrooms; in this way, 
groups of students completing each booklet are approximately equivalent in terms of student ability. 
PIRLS also employed this method of rotating reading passages and their associated items. The assignment 
design for TIMSS and PIRLS allowed collecting data across a broad subject content without 
overburdening the students. This is consistent with other large-scale assessments, such as the National 
Assessment of Educational Progress (NAEP). 


Test Administration 

Test administration for TIMSS and PIRLS in the United States occurred April through June 2011. The 
administration was carried out by professional staff trained according to the international guidelines. 
School personnel were asked only to assist with listings of students, the identification of space for testing 
in the school, and the specification of any parental consent procedures required. The International Study 
Center monitored compliance with the standardized procedures. 


Scoring 

Both the TIMSS and PIRLS assessment items included both multiple choice and constructed-response 
items. A scoring rubric (guide) was provided for every constructed response item. The National Research 
Coordinator in each country was responsible for the scoring and coding of data in that country, following 
established guidelines. 
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Scaling 


Total scores for mathematics and science in TIMSS, and reading in PIRLS, along with scores that reflect 
performance in specific domains of each subject, were estimated using an item response theory (IRT) 
model. For example, the TIMSS 2011 eighth-grade assessment had four scales describing four 
mathematics content areas and four science content areas, as well as three cognitive domains in each of 
mathematics and science. Benchmark scores were also derived. IRT estimation procedures were also used 
to place scores from the four TIMSS assessments conducted in 1995, 1999, 2003, 2007, and 2011 on the 
same scale (the scale of the 1995 administration). This allows for the calculation of trends in achievement 
even though the makeup of the countries participating in TIMSS changed over time. Details are provided 
in Martin and Mullis (2011). The same procedure was employed for PIRLS in which scores from the 
PIRLS assessments in 2006 and 2011 were placed onto the 2001 scale. 


Plausible Values 

The matrix sampling approach used in the TIMSS and PIRLS assessments meant that no student 
responded to all of the items. To accommodate the missing data generated by this design, during the 
scaling process, plausible values were estimated to characterize students participating in the assessment. 
Plausible values are imputed values and not test scores for individuals in the usual sense. They represent 
what the true performance of an individual might have been, had it been observed. They are estimated as 
random draws (usually five) from an empirically derived distribution of score values based on both the 
student’s observed responses to assessment items and on the student’s background variables. A more 
technical explanation can be found in Martin and Mullis (2011). From the point of view of analysis this 
means that each analysis has to be repeated five times, once for each plausible value, and the results 
averaged. 


Weighting 

Responses from the groups of students were assigned sampling weights to adjust for the complex sample 
design that resulted in students having an unequal, but known, probability of selection. Additionally, an 
adjustment for school and student nonresponse was built into the weighting. The estimation of sampling 
weights was carried out by Statistics Canada. A detailed description is provided in Martin and Mullis 
(2011). In analyses of the TIMSS and PIRLS data it is necessary to use sampling weights to obtain 
accurate population estimates. 
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1.5 


Reporting TIMSS and PIRLS Results 


Achievement results from TIMSS and PIRLS are reported on a scale from 0 to 1,000, with a scale average 
of 500 and standard deviation of 100. Even though the education systems participating in TIMSS and 
PIRLS have changed across the assessment rounds from the first administration (TIMSS in 1995 and 
PIRLS in 2001), comparisons between the 2011 results and prior results are still possible because the 
achievement scores in each of the assessments are placed on a scale that is not dependent on the list of 
participating education systems in any particular year. A more detailed explanation of the assessment’s 
equating and scaling can be found in Martin and Mullis (2011). 

In addition to numerical scale results, TIMSS and PIRLS also include international benchmarks. The 
international benchmarks provide a way to interpret the scale scores and to understand how students’ 
proficiency in a subject varies along the assessment scale. The benchmarks for both TIMSS and PIRLS 
describe four levels of student achievement in each subject, based on the kinds of skills and knowledge 
students at each score cut point would need to successfully answer the items. More information on the 
development of the benchmarks and the procedures used to set the score cut points can be found in Martin 
and Mullis (2011). 


1.6 U.S. International and National Data Files 


Three versions of the U.S. national data are available as follows: 

■ The TIMSS and PIRLS U.S. international data files that are available as part of the 
international database released by the International Study Center. The U.S. -specific 
TIMSS data files can be downloaded from 

http://timssandpirls.bc.edu/timss20 1 1/intemational-database.html. The U.S. -specific 
PIRLS data files are available separately and can be downloaded from 
http://timssandpirls.bc.edu/pirls201 1/intemational-database.html . These data files 
conform to the international specifications common to the data files from all countries. 
However, they do not include the U.S. -specific adaptations made to a few questions in 
the questionnaires or the additional questions added to the school and student 
questionnaires, such as the question on race/ethnicity added to the student 
questionnaire. Data for U.S. states are not included in the international data files due 
to potential confidentiality issues. 
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■ The TIMSS and PIRLS U.S. national public-use data files that are available 
through the National Center for Education Statistics. The TIMSS U.S. national dataset 
can be downloaded from http://nces.ed.gov/timss/datafiles.asp. The PIRLS U.S. 
national dataset can be downloaded from http://nces.ed.gov/survevs/pirls/datafiles.asp . 
These U.S. data files include (a) the U.S. -specific adaptations made to questionnaire 
items and (b) additional questions added to the school and student questionnaires. 

■ The TIMSS and PIRLS U.S. national restricted-use data files that are available 
through the National Center for Education Statistics. Access to these files may be 
obtained by completing a restricted-use license agreement with NCES. The restricted- 
use data files are provided only on CD. These data files contain supplemental link 
files that link TIMSS or PIRLS school ID numbers to the school ID numbers as they 
appear in the publicly available Common Core of Data (CCD) or the Private School 
Universe Survey (PSS). In addition, race/ethnicity is provided with all available 
categories and free or reduced-price lunch is provided as a continuous variable. 
Because these data can reveal the identities of participating schools, the restricted-use 
data files are only made available to those who obtain a NCES restricted-use data 
license. Directions on how to obtain the license can be found at 
http://nces.ed.gov/pubsearch/licenses.asp. 

State data are available for each of the benchmarking states via state -specific, restricted-use data files. 
Access to these files may be obtained by completing a restricted-use license agreement with NCES. These 
restricted-use data files are provided only on CD. They contain all of the data in the public released 
national data files plus the supplemental link files that link TIMSS or PIRLS school ID numbers to the 
school ID numbers as they appear in the publicly available Common Core of Data (CCD) or the Private 
School Universe Survey (PSS). Because these data can reveal the identities of participating schools, the 
restricted-use data files are only made available to those who obtain a NCES restricted-use data license. 
Directions on how to obtain the license can be found at http://nces.ed.gov/pubsearch/licenses.asp . 


The most comprehensive treatment of the TIMSS and PIRLS international data, and hence of the U.S. 
international data file, is provided in the various TIMSS 2011 and PIRLS 2011 publications produced by 
the International Association for the Evaluation of Educational Achievement (lEA), particularly the 
TIMSS 2011 User Guide for the International Database (Foy, Arora, & Stanco, 2013) and the PIRLS 
2011 User Guide for the International Database (Foy & Drucker, 2013). This publication should be seen 
as the primary reference. The TIMSS and PIRLS 2011 U.S. Technical Report and User Guide draws 
heavily on the international user’s guide for much of its data file-related content. This content is 
supplemented with detail on those aspects of the TIMSS 2011 and PIRLS 2011 data that were unique to 
the United States. 
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2 . 


Sampling Schools 


The 2011 TIMSS and PIRLS sample design consisted of the following samples: 

■ U.S. TIMSS fourth-grade national sample; 

■ U.S. PIRLS fourth-grade national sample; 

■ U.S. TIMSS eighth-grade national sample; 

■ NAEP -TIMSS eighth-grade national linking study sample; 

■ TIMSS fourth-grade state samples; 

■ PIRLS fourth-grade state sample; and 

■ TIMSS eighth-grade state benchmarking samples. 

Each of the first three pairs of samples are described jointly in this chapter because they shared the same 
sampled schools though they assessed separate classrooms within each school. 

All of the 2011 TIMSS and PIRLS school samples were drawn for the United States in March 2010. The 
sample design for the U.S. national school samples were developed to retain most of the properties of the 
2007 TIMSS and 2006 PIRLS U.S. school samples, and to follow international requirements as given in 
Martin and Mullis (2011). The U.S. samples again followed a two-stage sampling process with the first 
stage a sample of schools, and the second stage a sample of classrooms within schools. All students in 
sampled classrooms were selected for assessment. 


U.S. TIMSS and PIRLS Fourth-Grade National Samples 

The 2011 TIMSS and PIRLS national sample design for the fourth grade was similar to the sample design 
of TIMSS 2007 and PIRLS 2006 in that (a) the student population was defined as the set of all fourth- 
graders in the United States in both public and private schools, and (b) one or two classrooms were 
selected for each assessment in each sampled school. However, the 2011 TIMSS and PIRLS national 
sample design for the fourth grade differed from previous administrations of TIMSS and PIRLS in that 
there was one joint sample with up to four classrooms selected from each school. Schools in the sample 
with four or more fourth-grade classes had two classes randomly designated to either TIMSS or PIRLS, 
and the other two classes designated to the other study. In schools with three fourth-grade classes, two 
classes were randomly designated to one study, and the remaining class to the other study. In schools with 
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two fourth-grade classes, one class was randomly designated to one study, and the second class to the 
other study. In small schools with only one fourth-grade class, that class was randomly designated to 
either the TIMSS or PIRLS study. Thus, under this joint sampling design, the national fourth-grade 
sample contained 


■ schools with one classroom designated to TIMSS; 

■ schools with one classroom designated to PIRLS; and 

■ schools with two or more classrooms contributing classes to both TIMSS and PIRLS. 

In total, the 2011 fourth-grade national joint TIMSS-PIRLS school sample consisted of 450 schools 
containing at least one fourth-grade class. The schools were selected with probability proportionate to the 
school’s estimated grade enrollment of fourth-graders from the 2011 National Assessment of Educational 
Progress (NAEP) school frame using 2007-08 school data. The overall sample design was intended to 
approximate a self-weighting sample of students as much as possible in accordance with the international 
guidelines, with each fourth-grade student in the United States having an equal probability of being 
selected. 


U.S. TIMSS Eighth-Grade National and NAEP-TIMSS Eighth-Grade National 
Linking Study Samples 

The sample design for the eighth-grade national TIMSS assessment was also similar to the sample design 
of TIMSS 2007 in that the student population is the set of all eighth-graders in the United States in both 
public and private schools. However, unlike previous cycles where two classes per school were selected, 
only one class per school was selected in 201 1. This was due to a special NAEP-TIMSS linking study that 
was administered jointly in the same schools and that assessed a second classroom^ in a separate session 
from the TIMSS assessment. Thus, the TIMSS 2011 national school sample consisted of 600 schools with 
at least one eighth-grade class. The schools were selected with probability proportionate to the school’s 
estimated grade enrollment of eighth-graders from the 2011 NAEP school frame using 2007-08 school 
data. The overall sample design was intended to approximate a self-weighting sample of students as much 
as possible in accordance with the international guidelines, with each eighth-grade student in the United 
States having an equal probability of being selected. 


^Small schools with only one fourth-grade class would be designated to either the TIMSS or NAEP TIMSS Linking study. 
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TIMSS and PIRLS State Samples 


In 201 1, a PIRLS public school state sample was drawn for Florida; and a TIMSS public school state 
sample was drawn for Florida and North Carolina at both fourth- and eighth-grade and for Alabama, 
California, Colorado, Connecticut, Indiana, Massachusetts, and Minnesota at eighth grade only. The 2011 
TIMSS and PIRLS state samples replicated the national sample design, with the following differences: (a) 
there was only one joint sample at the fourth grade (in Florida), and (b) there was no NAEP-TIMSS 
linking study at the eighth grade in any of the benchmarking states. The school frame to draw the state 
samples was identical to the national frame of public schools in those states. The state samples included 
the schools in each state that were previously selected as part of the TIMSS and PIRLS national samples 
(these are sometimes referred to as “overlap” schools) plus a supplement of schools to reach the 
internationally set target of 100 assessed classrooms in each state. 


All Samples 

Schools were explicitly stratified by poverty level, public/private status, and census region. The schools 
were implicitly stratified by sorting by locale (city, suburb, town and rural), race/ethnicity enrollment 
(above or below 15 percent Black, Hispanic, Asian and Pacific Islander,^ and American Indian and 
Alaska Native), and the estimated grade enrollment within each explicit stratum. A systematic sample was 
selected independently in each stratum with sampling probabilities proportional to measures of size, 

(PPS) where the measure of size is the estimated number of students in fourth or eighth grade. 

There was no oversampling for either grade or study in 20 1 1 . Overlap with the 2011 NAEP school 
sample was minimized even though the TIMSS-PIRLS sample had to be selected before the NAEP 
sample because of TIMSS-PIRES scheduling constraints.^ The overlap between the samples was 
minimized when the 2011 NAEP sample was selected. 

The rest of this chapter describes the school samples. Section 2.1 describes the sampling frame — the data 
sources, data preparation, and the stratification variables. Section 2.2 describes the national school sample 


''Asian and Pacific Islander are combined into a single category due to the fact that that the frame was developed using the previous NCES 
guidelines for combining Asian and Pacific Islanders, while the U.S. national report was developed under new guidelines that call for reporting 
out Asian separately when possible. 

^In order to maximize response rates from both districts and schools, it was necessary to begin the recruitment of both prior to the end of the 
2009-10 school year. 
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selection — ^the measure of size, substitute schools, and selecting classrooms. Section 2.3 describes the 
state school sample selection. The data collection methods are covered in a chapter 4. 


2.1 School Sampling Frames 

The U.S. school sampling frames for fourth and eighth grade were developed from two national databases 
in the National Center for Education Statistics — public schools in the Common Core of Data (CCD 
http://nces.ed.gov/ccd/) and private schools in the Private School Survey (PSS 
http://nces.ed.gov/surveys/pss/) . These sources provide full coverage of all fourth- and eighth-grade 
students in the education system in the United States. The TIMSS-PIRLS 2011 fourth-grade school frame 
and the TIMSS 2011 eighth-grade school frame were constructed using the 2007-08 CCD and the 
2007-08 PSS, the most current data at the time of the frame construction. 

The data preparation of the school frames benefited from procedures developed for the National 
Assessment of Educational Progress (NAEP), a large educational survey in the United States. The NAEP 
method was described in the 1998 NAEP technical report (Allen, Donoghue, & Schoeps, 2001), and the 
most up-to-date information for each NAEP survey is available at http://nces.ed.gov/nationsreportcard/ . 
Additional NAEP reports are scheduled for release in the coming months. The school frames used the 
NAEP 2011 school frame as an input data source. 

Eligible schools in the school frames included schools operating in the 50 states and the District of 
Columbia, Department of Defense (DoD) domestic schools, and Bureau of Indian Education (BIE) 
schools. Schools in Puerto Rico and U.S. territories, DoD schools overseas, adult education institutions 
with no fourth- or eighth-grade students, and non-education institutions (e.g., home schools, 
correspondence schools) were ineligible for the study. 


Fourth-Grade Frame 

Any school having a fourth grade as of the school year 2007-08 was included on the TIMSS-PIRLS 
fourth-grade school sampling frame. Table 2-1 presents frame tabulations of the number of schools by the 
school grade span (lowest to highest grade level of the school). 
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Table 2-1. Number and percentage of students and schools included in the U.S. TIMSS-PIRLS fourth- 
grade school sampling frame, by grade span: 20 1 1 


Grade span 

Students 

Percent 

Schools 

Percent 

Total 

4,056,740 

100 

72,309 

100 

Grades 1-5 

2,007,383 

49.5 

26,317 

36.4 

Grades 1-6 

703,095 

17.3 

13,052 

18.1 

Grades 1-8 

464,896 

11.5 

15,297 

21.2 

Grades 1-12 

135,407 

3.4 

6,135 

8.5 

Other 

745,959 

18.4 

11,508 

15.9 


NOTE: The “other” grade spans includes all additional grade spans. Detail may not sum to totals because 


of rounding. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS); 2011. Progress in International Reading Literacy 
Study (PIRLS), 2011. 


Eighth-Grade Frame 


Any school having an eighth grade as of the school year 2007-08 was included on the TIMSS eighth- 
grade school sampling frame. Table 2-2 presents frame tabulations of the number of schools by the school 
grade span (lowest to highest grade level of the school). 


Table 2-2. Number and percentage of students and schools included in the U.S. TIMSS eighth-grade 
school sampling frame, by grade span: 2011 


Grade span 

Students 

Percent 

Schools 

Percent 

Total 

4,012,076 

100 

46,312 

100 

Grades 1 to 8 

464,914 

11.6 

15,297 

33.0 

Grades 6 to 8 

2,122,023 

52.9 

9,989 

21.6 

Grades 1 to 12 

135,447 

3.4 

6,135 

13.3 

Grades 7 to 12 

143,229 

3.6 

3,213 

6.9 

Grades 7 to 8 

686,791 

17.1 

2,878 

6.2 

Other 

459,672 

11.5 

8,800 

19.0 


NOTE: Detail may not sum to totals because of rounding. The “Other” grade spans includes all 
additional grade spans. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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stratification 


The sample design was a stratified systematic sample within each stratum, with sampling probabilities 
proportional to size (PPS). Stratification was used for sample efficiency and consistency with previous 
designs. The explicit strata were formed by the following variables, shown in alphabetical order: 


■ Census region — Northeast, Midwest, South, and West; 

■ poverty level'^ — for public schools, “high” poverty is defined as having 50 percent or 
more of the students eligible for participation in the free or reduced-price lunch 
program (FRPL), and “low” poverty is defined as having less than 50 percent eligible; 
and 

■ school type — school is either under public control (operated by publicly elected or 
appointed officials) or private control (operated by privately elected or appointed 
officials and derives its major source of funds from private sources). 


Within each stratum, the frame was implicitly stratified by the following three categorical stratification 
variables: 


■ locale — urban-centric locale code, i.e., city, suburb, town, rural; 

■ race/ethnicity status — student population in the school is “15 percent or above” or 
“below 1 5 percenf ’ Black, Hispanic, Asian and Pacific Islander, and American Indian 
and Alaska Native students;’ and 

■ estimated grade enrollment. 


Fourth-Grade Stratification 

The following tables show the total number and percentage of fourth-grade students and schools in the 
TIMSS-PIRLS 2011 school frame by census region (table 2-3), poverty level (table 2-4), school type 
(table 2-5), locale (table 2-6), race/ethnicity status (table 2-7), and by poverty level, school type and 
region (table 2-8). 


^The sample frame did not contain a direct measure of poverty. No National School Lunch Program (NSLP) data were available for private 
schools, thus all private schools are treated as low-poverty schools. 

^Black includes African American and Hispanic includes Latino. Racial categories exclude Hispanic origin. Asian and Pacific Islander are 
combined into a single category due to the fact that that the frame was developed using the previous NCES guidelines for combining Asian and 
Pacific Islanders, while the U.S. national report was developed under new guidelines that call for reporting out Asian separately when possible. 
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Table 2-3. Number and percentage of students and schools included in the U.S. TIMSS-PIRLS fourth- 
grade school sampling frame, by region: 2011 


Region 

Students 

Percent 

Schools 

Percent 

Total 

4,056,740 

100 

72,309 

100 

Northeast 

670,506 

16.5 

12,824 

17.7 

Midwest 

885,979 

21.8 

18,605 

25.7 

South 

1,537,165 

37.9 

23,713 

32.8 

West 

963,090 

23.7 

17,167 

23.8 


NOTE: Detail may not sum to totals because of rounding. Region of country is based on Census 
definitions. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 


Table 2-4. Number and percentage of students and schools included in the U.S. TlMSS-PlRLS fourth- 
grade school sampling frame, by poverty level: 2011 


Poverty level 

Students 

Percent 

Schools 

Percent 

Total 

4,056,740 

100 

72,309 

100 

High 

1,610,082 

39.7 

23,166 

32.0 

Low 

2,446,658 

60.3 

49,143 

68.0 


NOTE: For public schools, “high” poverty is defined as having 50 percent or more of the students 
eligible for participation in the National School Lunch Program (NSLP), and “low” poverty is defined as 
having less than 50 percent eligible. Because no NSLP data were available for private schools, all private 
schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 


Table 2-5. Number and percentage of students and schools included in the U.S. TIMSS-PIRLS fourth- 
grade school sampling frame, by school type: 2011 


School type 

Students 

Percent 

Schools 

Percent 

Total 

4,056,740 

100 

72,309 

100 

Private 

387,559 

9.6 

20,852 

28.8 

Public 

3,669,181 

90.4 

51,457 

71.2 


SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 
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Table 2-6. Number and percentage of students and schools included in the U.S. TIMSS-PIRLS fourth- 
grade school sampling frame, by locale: 2011 


Locale 

Students 

Percent 

Schools 

Percent 

Total 

4,056,740 

100 

72,309 

100 

City 

1,235,518 

30.5 

21,879 

30.3 

Suburb 

1,457,038 

35.9 

22,254 

30.8 

Town 

479,595 

11.8 

8,184 

11.3 

Rural 

884,589 

21.8 

19,992 

27.6 


NOTE: For definitions of these urban-centric locales, see 
http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 


Table 2-7. Number and percentage of students and schools included in the U.S. TIMSS-PIRLS fourth- 
grade school sampling frame, by race/ethnicity status: 2011 


Race/ethnicity status 

Students 

Percent 

Schools 

Percent 

Total 

4,056,740 

100 

72,309 

100 

1 5 percent or above 

2,535,765 

62.5 

39,341 

54.4 

Below 1 5 percent 

1,520,975 

37.5 

32,968 

45.6 


NOTE: Race/ethnicity status refers to the percentage of Black, Hispanic, Asian and Pacific Islander, and 
American Indian and Alaska Native students. Black includes African American and Hispanic includes 
Latino. Racial categories exclude Hispanic origin. Asian and Pacific Islander are combined into a single 
category due to the fact that that the frame was developed using the previous NCES guidelines for 
combining Asian and Pacific Islanders, while the U.S. national report was developed under new 
guidelines that call for reporting out Asian separately when possible. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in Intemational Reading Literacy 
Study (PIRLS), 2011. 
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Table 2-8. Number and percentage of students and schools included in the U.S. TIMSS-PIRLS fourth- 
grade school sampling frame, by region, poverty level, and school type: 2011 


Region 

Poverty level 

School type 

Students 

Percent 

Schools 

Percent 

Total 



4,056,740 

100 

72,309 

100 

Northeast 

High 

Public 

183,987 

4.5 

2,647 

3.7 

Midwest 

High 

Public 

224,192 

5.5 

4,153 

5.7 

South 

High 

Public 

761,830 

18.8 

10,205 

14.1 

West 

High 

Public 

440,073 

10.8 

6,161 

8.5 

Northeast 

Low 

Private 

87,838 

2.2 

4,701 

6.5 

Midwest 

Low 

Private 

97,418 

2.4 

5,426 

7.5 

South 

Low 

Private 

126,634 

3.1 

6,491 

9.0 

West 

Low 

Private 

75,669 

1.9 

4,234 

5.9 

Northeast 

Low 

Public 

398,681 

9.8 

5,476 

7.6 

Midwest 

Low 

Public 

564,369 

13.9 

9,026 

12.5 

South 

Low 

Public 

648,701 

16.0 

7,017 

9.7 

West 

Low 

Public 

447,348 

11.0 

6,772 

9.4 


NOTE: Region of country is based on Census definitions. For public schools, “high” poverty is defined 
as having 50 percent or more of the students eligible for participation in the National School Lunch 
Program (NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP 
data were available for private schools, all private schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 


Eighth-Grade Stratification 

The following tables show the total number and percentage of eighth grade students and schools in the 
TIMSS 2011 eighth-grade school frame by Census region (table 2-9), poverty level (table 2-10), school 
type (table 2-1 1), locale (table 2-12), race/ethnicity status (table 2-13), and by poverty level, school type, 
and region (table 2-14). 
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Table 2-9. Number and percentage of students and schools included in the U.S. TIMSS eighth-grade 
school sampling frame, by region: 2011 


Region 

Students 

Percent 

Schools 

Percent 

Total 

4,012,076 

100 

46,312 

100 

Northeast 

687,958 

17.1 

8,745 

18.9 

Midwest 

892,364 

22.2 

12,546 

27.1 

South 

1,479,254 

36.9 

14,735 

31.8 

West 

952,500 

23.8 

10,286 

22.2 


NOTE: Region of country is based on Census definitions. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 2-10. Number and percentage of students and schools included in the U.S. TIMSS eighth-grade 
school sampling frame, by poverty level: 2011 


Poverty level 

Students 

Percent 

Schools 

Percent 

Total 

4,012,076 

100 

46,312 

100 

High 

1,353,825 

33.7 

11,345 

24.5 

Low 

2,658,251 

66.3 

34,967 

75.5 


NOTE: For public schools, “high” poverty is defined as having 50 percent or more of the students 
eligible for participation in the National School Lunch Program (NSLP), and “low” poverty is defined as 
having less than 50 percent eligible. Because no NSLP data were available for private schools, all private 
schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 2-11. Number and percentage of students and schools included in the U.S. TIMSS eighth-grade 
school sampling frame, by school type: 2011 


School type 

Students 

Percent 

Schools 

Percent 

Total 

4,012,076 

100 

46,312 

100 

Private 

373,091 

9.3 

18,710 

40.4 

Public 

3,638,985 

90.7 

27,602 

59.6 


SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 2-12. Number and percentage of students and schools included in the U.S. TIMSS eighth-grade 
school sampling frame, by locale: 2011 


Locale 

Students 

Percent 

Schools 

Percent 

Total 

4,012,076 

100 

46,312 

100 

City 

1,190,107 

29.7 

13,134 

28.4 

Suburb 

1,434,200 

35.7 

12,524 

27.0 

Town 

496,556 

12.4 

5,560 

12.0 

Rural 

891,213 

22.2 

15,094 

32.6 


NOTE: For definitions of these urban-centric locales, see 
http://nces.ed.gov/survevs/urbaned/definitions.asp . 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 2-13. Number and percentage of students and schools included in the U.S. TIMSS eighth-grade 
school sampling frame, by race/ethnicity status: 2011 


Race/ethnicity status 

Students 

Percent 

Schools 

Percent 

Total 

4,012,076 

100 

46,312 

100 

1 5 percent or above 

2,518,075 

62.8 

23,372 

50.5 

Below 1 5 percent 

1,494,001 

37.2 

22,940 

49.5 


NOTE: Race/ethnicity status refers to the percentage of Black, Hispanic, Asian and Pacific Islander, and 
American Indian and Alaska Native students. Black includes African American and Hispanic includes 
Latino. Racial categories exclude Hispanic origin. Asian and Pacific Islander are combined into a single 
category due to the fact that that the frame was developed using the previous NCES guidelines for 
combining Asian and Pacific Islanders, while the U.S. national report was developed under new 
guidelines that call for reporting out Asian separately when possible. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 2-14. Number and percentage of students and schools included in the U.S. TIMSS eighth-grade 
school sampling frame, by region, poverty level, and school type: 2011 


Region 

Poverty level 

School type 

Students 

Percent 

Schools 

Percent 

Total 



4,012,076 

100 

46,312 

100 

Northeast 

High 

Public 

158,560 

4.0 

1,398 

3.0 

Midwest 

High 

Public 

180,246 

4.5 

2,175 

4.7 

South 

High 

Public 

628,262 

15.7 

4,990 

10.8 

West 

High 

Public 

386,757 

9.6 

2,782 

6.0 

Northeast 

Low 

Private 

88,216 

2.2 

4,523 

9.8 

Midwest 

Low 

Private 

91,422 

2.3 

4,708 

10.2 

South 

Low 

Private 

121,053 

3.0 

5,713 

12.3 

West 

Low 

Private 

72,400 

1.8 

3,766 

8.1 

Northeast 

Low 

Public 

441,182 

11.0 

2,824 

6.1 

Midwest 

Low 

Public 

620,696 

15.5 

5,663 

12.2 

South 

Low 

Public 

729,939 

18.2 

4,032 

8.7 

West 

Low 

Public 

493,343 

12.3 

3,738 

8.1 


NOTE: Detail may not sum to totals because of rounding. Region of country is based on Census 
definitions. For public schools, “high” poverty is defined as having 50 percent or more of the students 
eligible for participation in the National School Lunch Program (NSLP), and “low” poverty is defined as 
having less than 50 percent eligible. Because no NSLP data were available for private schools, all private 
schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


2.2 National School Samples 

The U.S. national samples used a two-stage design — a stratified systematic sample of schools with 
sampling probabilities proportional to size (PPS) and then classes within sampled schools. The school 
selection probability was configured such that all fourth- or eighth-grade students in the United States 
would have approximately equal probability of being selected in the samples. A sample of 450 schools 
was drawn from the fourth-grade frame; and a sample of 600 schools was drawn from the eighth-grade 
frame. 

Within each explicit stratum, the frames were implicitly stratified by two categorical stratification 
variables. The order of the stratification is not given because of confidentiality concerns. Each frame was 
sorted in alternating (serpentine) sort order according to these school characteristics, implicitly stratifying 
the frame. The last sort within the implicit stratification was by grade enrollment (measure of size or 
MOS) in descending order. Alternating the “sort order” sorts a frame from lowest to highest value with 
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respect to the first sort variable, then within each level of the first sort variable, the second sort variable 
alternates its sort order, from lowest to highest for the first level of the first sort variable, then from 
highest to lowest for the second level of the first sort variable, then again from lowest to highest for the 
third level of the first sort variable, and so on. Each of the variables will alternate the sort order within 
each level of the preceding sort variable. This means that schools adjacent on the list are not substantially 
different or at most different by one sorting characteristic. 


Measure of Size 

The goal for the TIMSS-PIRLS sample was to attain a self-weighting student sample. To achieve this, 
schools’ probability of selection was related to their measure of size (MOS), which is proportional to its 
share of the target population, that is, the fourth- or eighth-grade enrollments. This method also reduces 
the chance of selection for smaller schools. This improves cost efficiency by increasing the number of 
students per school. However, students in schools with enrollments of only a few students would have 
very large weights if selected. To minimi z e the impact of these small schools on variances and estimates, 
the minimum measure of size was set to 5. 

The following is a summary of the steps for assigning measures of size to the schools on the TIMSS and 
PIRLS frames. 

Determine the estimated target population size for the school. This is the estimated enrollment per grade 
(4 or 8) from the school frame. If the grade 4 or 8 enrollment is not available, it is calculated by dividing 
the school’s total enrollment by the number of grades in the school. 

Calculate measures of size according to the estimated enrollment per grade as shown: 

fs if grade 4 or 8 enrollment <= 5 

MOS = 

[grade 4 or 8 enrollment otherwise 


Substitute Schools 

Although efforts were made to secure the participation of all schools selected, it was anticipated at the 
time of sampling that not all schools would choose to participate. Therefore, as each school was selected 
for a sample, the two neighboring schools in the sampling frame were designated as substitute schools. 
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The first school following the sample school was the first substitute, and the first school preceding it was 
the second substitute. If an original school refused to participate, the first substitute was then contacted. If 
that school also refused to participate, the second substitute was then contacted. 

There were several constraints on the assignment of substitutes. One sampled school was not allowed to 
be a substitute for another, and a given school could not be assigned to be a substitute for more than one 
sampled school. Furthermore, substitutes were required to be in the explicit stratum as the sampled 
school. If the sampled school was the first or last school in the stratum, then the second school following 
or preceding the sampled school was identified as the substitute. If the first substitute school did not have 
the same implicit stratification values as the sampled school, the first and second substitute were 
switched. Under these rules, it was possible to identify two substitutes for each sampled school. 


Selecting Classrooms 

The final stage of selection was of classrooms within schools. Within each sampled school that agreed to 
participate in TIMSS and PIRLS at fourth grade, all classrooms in the school were listed on the classroom 
sampling frame. Within each sampled school that agreed to participate in TIMSS at eighth grade, all 
mathematics classrooms in the school were listed on the classroom sampling frame. Schools were asked 
to indicate the names of all classes containing fourth- or eighth-grade students, the number of fourth- or 
eighth-grade students in the class, and whether it was a “special class” in which all or most of the students 
were learning disabled or classified as having limited English proficiency. Because TIMSS and PIRLS do 
not provide accommodations, classrooms were excluded from the subsequent classroom sampling if all or 
most of the students were learning disabled. Classrooms with fewer than 1 5 students were collapsed into 
pseudoclassrooms so that each classroom on the school’s classroom sampling frame had at least 20 
students.* An equal probability sample of classrooms or pseudoclassrooms was sampled from the 
classroom frame for each school. All students in sampled classrooms (pseudoclassrooms) were selected 
for assessment. 

Schools in the sample with four or more fourth-grade classes had two classes randomly designated to 
either TIMSS or PIRLS and the other two designated to the other study. In schools with three fourth- 
grade classes, two classes were randomly designated to one study, and the remaining class to the other 


*Pseudoclasses were automatically created during the sampling of classes if there were small classes in a school. A small class was defined as a 
class for which the number of students was smaller than half the average class size in the explicit stratum across all schools. In such a case, 
WinW3S will automatically sampled two or more classes that were grouped together to form a pseudoclass. 


U.S. TIMSS and PIRLS 2011 Technical Report and User ’s Guide 


22 



study. In schools with two fourth-grade classes, one class was randomly designated to one study and the 
second class to the other study. In small schools with only one fourth-grade class, that class was randomly 
designated to either the TIMSS or PIRLS study. Thus, under this sampling design, the three groups of 
schools samples are as follows: 

■ schools with one classroom designated to TIMSS, 

■ schools with one classroom designated to PIRLS, and 

■ schools with two or more classrooms contributing classes to both TIMSS and PIRLS. 

Schools of two or more eighth-grade classes were identified in the sample, and classes in those schools 
were randomly designated to either TIMSS or the NAEP-TIMSS linking study. Schools with only one 
eighth-grade class were randomly assigned to either TIMSS or the NAEP-TIMSS linking study. 


Tabulations Within Subgroups for Frame and Sample 

This section provides an overview of the frame and sample for the explicit and implicit strata used in the 
sample process. The PPS sampling and stratification worked effectively: the sample percentage of schools 
is close to the measure-of-size percentage of the frame for all the implicit strata. For these strata-defming 
subgroups, tables 2-15 through 2-26 present the following summary tabulations in these subgroups: 

■ Total measure of size. This is the summation of MOSy over the subgroup. Note that 
this is larger than the national population student size because the minimum MOSy is 
set to 5 for small schools; and 

■ Sample size. This is the final realized sample size of schools in the subgroup for the 
U.S. TIMSS-PIRES fourth- or eighth-grade samples. 


Fourth-Grade Tabulations 

This section provides an overview of the fourth-grade frame and sample distribution by each of the 
stratification variables. Each table shows the total number and percentage of fourth-grade students in the 
sampling frame (data shown in tables 2-15 through 2-20) and the total number and percentage of schools 
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in the TIMSS-PIRLS school sample.^ By each stratification variable, the tables are Census region 
(table 2-15), poverty level (table 2-16), school type (table 2-17), locale (table 2-18), race/ethnicity status 
(table 2-19), and by poverty level, school type, and region (table 2-20). 


Table 2-15. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TlMSS-PlRLS fourth-grade, by region: 2011 


Region 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,067,306 

100 

450 

100 

Northeast 

672,435 

16.5 

74 

16.4 

Midwest 

888,557 

21.8 

98 

21.8 

South 

1,540,111 

37.9 

170 

37.8 

West 

966,203 

23.8 

108 

24.0 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. Region of country is based on Census definitions. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Eiteracy 
Study (PIRLS), 2011. 


Table 2-16. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS-PIRLS fourth-grade, by poverty level: 2011 


Poverty level 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,067,306 

100 

450 

100 

High 

1,611,039 

39.6 

178 

39.6 

Low 

2,456,267 

60.4 

272 

60.4 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. For public schools, “high” poverty is defined as having 50 percent or more of the 
students eligible for participation in the National School Lunch Program (NSLP), and “low” poverty is 
defined as having less than 50 percent eligible. Because no NSLP data were available for private schools, 
all private schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 


’The measure of size (MOS) defined as the estimated number of students enrolled in the target grade with a minimum of 5 students per school. 
These are consistently larger than the estimated student sample size (reported in tables 2.1-2.14), which is the estimate of the number of students 
in the sampled schools and have no minimum per school. 
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Table 2-17. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS-PIRLS fourth-grade, by school type: 2011 


School type 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,067,306 

100 

450 

100 

Private 

395,483 

9.7 

44 

9.8 

Public 

3,671,823 

90.3 

406 

90.2 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Eiteracy 
Study (PIRES), 2011. 


Table 2-18. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS-PIRES fourth-grade, by locale: 2011 


Eocale 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,067,306 

100 

450 

100 

City 

1,237,410 

30.4 

138 

30.7 

Suburb 

1,459,072 

35.9 

161 

35.8 

Town 

480,958 

11.8 

51 

11.3 

Rural 

889,866 

21.9 

100 

22.2 


NOTE: For definitions of these urban-centric locales, see 

http://nces.ed.gov/survevs/urbaned/definitions.asp . Measure of size is the estimated number of students 
enrolled in the target grade with a minimum of 5 students per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in Intemational Reading Eiteracy 
Study (PIRES), 2011. 
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Table 2-19. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS-PIRLS fourth-grade, by race/ethnicity status: 2011 


Race/ethnicity status 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,067,306 

100 

450 

100 

1 5 percent or above 

2,540,199 

62.5 

278 

61.8 

Below 1 5 percent 

1,527,107 

37.5 

172 

38.2 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. Race/ethnicity status refers to the percentage of Black, Hispanic, Asian and 
Pacific Islander, and American Indian and Alaska Native students. Black includes African American and 
Hispanic includes Latino. Racial categories exclude Hispanic origin. Asian and Pacific Islander are 
combined into a single category due to the fact that that the frame was developed using the previous 
NCES guidelines for combining Asian and Pacific Islanders, while the U.S. national report was 
developed under new guidelines that call for reporting out Asian separately when possible. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 
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Table 2-20. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS-PIRLS fourth-grade, by region, poverty level, and 
school type: 2011 


Region 

Poverty level 


Frame 


Sample 


School type 

Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



4,067,306 

100 

450 

100 

Northeast 

High 

Public 

184,025 

4.5 

20 

4.4 

Midwest 

High 

Public 

224,455 

5.5 

25 

5.6 

South 

High 

Public 

761,980 

18.7 

84 

18.7 

West 

High 

Public 

440,579 

10.8 

49 

10.9 

Northeast 

Low 

Private 

89,664 

2.2 

10 

2.2 

Midwest 

Low 

Private 

99,233 

2.4 

11 

2.4 

South 

Low 

Private 

129,185 

3.2 

14 

3.1 

West 

Low 

Private 

77,401 

1.9 

9 

2.0 

Northeast 

Low 

Public 

398,746 

9.8 

44 

9.8 

Midwest 

Low 

Public 

564,869 

13.9 

62 

13.8 

South 

Low 

Public 

648,946 

16.0 

72 

16.0 

West 

Low 

Public 

448,223 

11.0 

50 

11.1 


NOTE: Detail may not sum to totals because of rounding. Measure of size is the estimated number of 
students enrolled in the target grade with a minimum of 5 students per school. Region of country is based 
on Census definitions. For public schools, “high” poverty is defined as having 50 percent or more of the 
students eligible for participation in the National School Lunch Program (NSLP), and “low” poverty is 
defined as having less than 50 percent eligible. Because no NSLP data were available for private schools, 
all private schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011; Progress in International Reading Literacy 
Study (PIRLS), 2011. 


Eighth-Grade Tabulations 


This section provides an overview of the eighth-grade frame and sample distribution by each of the 
stratification variables. Each table shows the total number and percentage of eighth-grade students in the 
sampling frame (data shown in tables 2-2 1 through 2-26 and the total number and percentage of schools 
in the TIMSS eighth-grade school sample. By each stratification variable, the tables are Census region 
(table 2-21), poverty level (table 2-22), school type (table 2-23), locale (table 2-24), race/ethnicity status 
(table 2-25), and by poverty level, school type, and region (table 2-26). 
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Table 2-21. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS eighth-grade, by region: 2011 


Region 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,024,167 

100 

600 

100 

Northeast 

690,035 

17.1 

103 

17.2 

Midwest 

895,169 

22.2 

133 

22.2 

South 

1,482,889 

36.9 

221 

36.8 

West 

956,074 

23.8 

143 

23.8 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. Region of country is based on Census definitions. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 2-22. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS eighth-grade, by poverty level: 2011 


Poverty level 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,024,167 

100 

600 

100 

High 

1,355,600 

33.7 

203 

33.8 

Eow 

2,668,567 

66.3 

397 

66.2 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. For public schools, “high” poverty is defined as having 50 percent or more of the 
students eligible for participation in the National School Eunch Program (NSEP), and “low” poverty is 
defined as having less than 50 percent eligible. Because no NSEP data were available for private schools, 
all private schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 2-23. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS eighth-grade, by school type: 2011 


School type 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,024,167 

100 

600 

100 

Private 

380,980 

9.5 

56 

9.3 

Public 

3,643,187 

90.5 

544 

90.7 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 2-24. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS eighth-grade, by locale: 2011 


Eocale 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,024,167 

100 

600 

100 

City 

1,192,531 

29.6 

179 

29.8 

Suburb 

1,436,705 

35.7 

213 

35.5 

Town 

498,424 

12.4 

75 

12.5 

Rural 

896,507 

22.3 

133 

22.2 


NOTE: For definitions of these urban-centric locales, see 

http://nces.ed.gov/survevs/urbaned/definitions.asp . Measure of size is the estimated number of students 
enrolled in the target grade with a minimum of 5 students per school. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 2-25. Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS eighth-grade, by race/ethnicity status: 2011 


Race/ethnicity status 

Frame 


Sample 


Measure of size 

Percent 

Number of schools 

Percent 

Total 

4,024,167 

100 

600 

100 

1 5 percent or above 

2,524,187 

62.7 

378 

63.0 

Below 1 5 percent 

1,499,980 

37.3 

222 

37.0 


NOTE: Measure of size is the estimated number of students enrolled in the target grade with a minimum 
of 5 students per school. Race/ethnicity status refers to the percentage of Black, Hispanic, Asian and 
Pacific Islander, and American Indian and Alaska Native students. Black includes African American and 
Hispanic includes Eatino. Racial categories exclude Hispanic origin. Asian and Pacific Islander are 
combined into a single category due to the fact that that the frame was developed using the previous 
NCES guidelines for combining Asian and Pacific Islanders, while the U.S. national report was 
developed under new guidelines that call for reporting out Asian separately when possible. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 2-26. 


Number and percentage of students in the sampling frame and number and percentage of 
schools in the sample, U.S. TIMSS eighth-grade, by region, poverty level, and school type: 


2011 


Region 

Poverty level 


Frame 


Sample 


School type 

Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



4,024,167 

100 

600 

100 

Northeast 

High 

Public 

158,633 

3.9 

24 

4.0 

Midwest 

High 

Public 

180,697 

4.5 

27 

4.5 

South 

High 

Public 

628,781 

15.6 

94 

15.7 

West 

High 

Public 

387,489 

9.6 

58 

9.7 

Northeast 

Low 

Private 

90,144 

2.2 

13 

2.2 

Midwest 

Low 

Private 

93,092 

2.3 

14 

2.3 

South 

Low 

Private 

123,526 

3.1 

18 

3.0 

West 

Low 

Private 

74,218 

1.8 

11 

1.8 

Northeast 

Low 

Public 

441,258 

11 

66 

11.0 

Midwest 

Low 

Public 

621,380 

15.4 

92 

15.3 

South 

Low 

Public 

730,582 

18.2 

109 

18.2 

West 

Low 

Public 

494,367 

12.3 

74 

12.3 


NOTE: Detail may not sum to totals because of rounding. Measure of size is the estimated number of 
students enrolled in the target grade with a minimum of 5 students per school. Region of country is based 
on Census definitions. For public schools, “high” poverty is defined as having 50 percent or more of the 
students eligible for participation in the National School Lunch Program (NSLP), and “low” poverty is 
defined as having less than 50 percent eligible. Because no NSLP data were available for private schools, 
all private schools are categorized as “low.” 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


2.3 State School Samples 

The state school samples were selected only for public schools as follows: 

■ TIMSS fourth-grade sample - North Carolina; 

■ TIMSS and PIRLS fourth-grade samples - Florida; and 

■ TIMSS eighth-grade benchmarking samples - Alabama, California, Colorado, 
Connecticut, Florida, Indiana, Massachusetts, Minnesota, and North Carolina. 
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The school frame used to draw the state samples was identical to the national frame of public schools in 
those states and grades.'® However, the North Carolina fourth-grade state school selection differed from 
that for Florida because North Carolina did not participate in PIRLS. Thus, its fourth-grade sample was 
completely independent and did not include the schools in the state that were selected for the national 
sample (overlap schools). In contrast, Florida’s fourth-grade state sample and the eighth-grade 
benchmarking state samples were based on an integrated state and national design in which schools in the 
state that were selected for the national sample were also selected for the state sample. For each state, 
tables showing its frame and sample distribution, by each of the stratification variables, are provided in 
appendix A. 

The rest of the design was similar to the national design where possible. There were only two explicit 
strata per state (high/low poverty level), as the other national strata did not apply to state samples. As with 
the national samples, the frame was implicitly stratified by location, race/ethnicity enrollment, and 
estimated grade enrollment. In addition, it was also implicitly stratified by state achievement scores." The 
MOS for each school was the same as in the national design. Substitute schools were assigned using the 
same procedure. 

The procedure for selecting classrooms was also the same as in the national design. However, both classes 
were assigned to the state sample in schools with two or more classes for the fourth-grade North Carolina 
and eighth-grade benchmarking samples, while the classroom designation was the same as the national 
design for the fourth-grade Florida sample. 


School Selection for the Integrated Design 

The state samples for Alabama, California, Colorado, Connecticut, Florida, Indiana, Massachusetts, and 
Minnesota, and North Carolina at eighth-grade used an integrated design. As mentioned above, the 
fourth-grade sample in North Carolina differed in that it was independent. The integrated design included 
the schools in each state that were previously selected as part of the TIMSS and PIRLS national samples 
plus a supplement of schools to reach the international target of 100 assessed classrooms. The target was 
in terms of the number of classrooms rather than schools because with the integration of the national 
design there were a varying number of national sample schools per state. For the eighth-grade samples. 


'”For this reason, tables are not provided showing the frame distributions for each benchmarking state by each of the respective stratification 
variables. Tables showing the each state’s frame and sample distribution by each of the stratification variables are provided in appendix A. 

^^The state achievement scores were derived for NAEP from the results of the state assessment programs obtained from each state. 
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one class was selected for TIMSS and one for the NAEP-TIMSS linking study in each school. Thus, only 
one classroom from each of the national schools was included in the state assessment. The supplemental 
grade 8 sample of schools selected for each state followed the normal TIMSS procedure of selecting two 
classes per school. The additional number of schools needed in each state is then ([100 - # national public 
schools] / 2) plus an additional five schools per state to account for ineligible schools. The supplemental 
fourth-grade Florida sample followed the TIMSS-PIRLS procedure of selecting four classes per school, 
so an additional 20 supplemental schools were selected to account for schools with fewer than four 
classrooms. 

The integrated sample was selected using a version of the Keyfitz procedure (Keyfitz, 1951); Chowdhury, 
Chu, & Kaufman, (2001) have described the implementation of the procedure. The method is generally 
used to minimize overlap between one or more surveys, but it can also be used to maximize overlap by 
ordering the rows in descending order of the response load indicator. By following the process outlined in 
table 2 of the Chowdhury, Chu, and Kaufman paper, the rows in the table can be thought of as a hierarchy 
of selection preference, where the top row maximizes the probability and the bottom row minimizes it. 
This property allowed us to maximize the overlap with the TIMSS national sample (in fact, select all 
national schools) while minimizing the overlap with the NAEP state operational public school sample or 
“Alpha sample.”'^ By maximizing the overlap with the national sample, the assessed classrooms can be 
included in both studies with proper probabilities. This minimization was undertaken to reduce the burden 
for schools selected in the NAEP Alpha sample and to improve response rates. This was accomplished by 
partitioning the frame into the following three groups shown in order, as in table 2 of the paper. The three 
groups were as follows: 

■ schools selected for the TIMSS national sample (including schools also selected for 
the NAEP Alpha sample); 

■ schools not selected for either the TIMSS national sample or the NAEP Alpha sample; 
and 

■ schools selected for the NAEP Alpha sample and not for the TIMSS national sample. 

With this design, the method guarantees all schools in group 1 will be selected with certainty since the 
probability of being selected for the integrated sample is always larger than being selected for the national 
sample and since more schools were selected in each state sample (the national schools plus a state 
supplement) than in the national sample with the frames being identical. The method minimized the 


'^For a complete definition of the NAEP Alpha sample, see the NCES NAEP glossary at 
http://nces.ed.gOv/nationsreportcard/glossary.asp#alpha sample . 
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overlap with schools in group 3 (NAEP Alpha sample) and selected the majority of the state supplement 
from schools in group 2. Table 2-27 presents the composition of the state integrated sample by sample 
type and state. See appendix A for the tables of the frame and sample distribution by each of the 
stratification variables for each state sample. 


Table 2-27. Number of schools included in the state samples, by grade/design, state, and sample type: 
2011 


Grade/design 

State 

Number of schools 
in national sample 

Number of schools 
in state supplement 
sample 

Number of schools 
in state integrated 
sample 

Fourth (independent) 

NC 

0 

49 

49 

F ourth-integrated 

EL 

21 

60 

81 

Eighth-integrated 

AL 

12 

51 

63 


CA 

73 

21 

94 


CO 

10 

50 

60 


CT 

11 

52 

63 


EL 

21 

44 

65 


IN 

10 

52 

62 


MA 

8 

50 

58 


MN 

8 

52 

60 


NC 

15 

47 

62 


SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


School Selection for the North Carolina Fourth-Grade Design 

Ideally, the objective for the North Carolina state samples was that they would not include the schools 
that were previously selected as part of the TIMSS national sample. By again following the Keyfitz 
procedure outlined in table 2 of Chowdhury, Chu, and Kaufman (2001), the procedure allowed us to 
minimize the overlap with the TIMSS national and NAEP Alpha (public school) samples. By minimizing 
the overlap with the national sample, the assessed classrooms can be included in only one study with 
proper probabilities. This was accomplished by partitioning the frame into the following three groups 
shown in order as in table 2 of the paper. The four groups were as follows: 

■ schools not selected for either the TIMSS national or NAEP Alpha samples; 

■ schools selected for the NAEP Alpha sample and not the TIMSS national sample; 
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■ schools selected for the TIMSS national sample and not the NAEP Alpha sample; and 

■ schools selected for the TIMSS national sample and the NAEP Alpha sample. 

With this design, the method accomplished the goal of selecting all the state sample from group 1 and 
none from the other groups. For the schools in groups 3 and 4, this was due to the sum of the school’s 
probabilities of being selected for the state sample and the national sample was always less than one. In 
that case, their conditional probabilities are zero. 
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3 . 


Participation Rates and Nonresponse Bias 


To mi ni miz e the potential for response biases, lEA developed participation or response rate standards that 
apply to all participating education systems and govern whether or not data are included in the TIMSS 
and PIRLS 2011 international datasets and the way in which aggregate statistics are presented in the 
international reports. These standards were set using composites of participation rates at the school, 
classroom, and student levels, and were calculated with and without the inclusion of substitute schools 
that were selected to replace schools refusing to participate. 

The standards take the following two forms, distinguished primarily by whether or not meeting the school 
participation rate of 85 percent requires the counting of substitute schools: 

Category 1: Met requirements. Participants that meet all of the following conditions are considered to 
have fulfilled the lEA requirements: 

Obtain an unweighted school response rate of at least 85 percent without replacement (rounded to nearest 
whole percent) AND an unweighted student response rate (after rounding) of at least 85 percent 

OR 

A weighted school response rate of at least 85 percent without replacement (rounded to nearest whole 
percent) AND a weighted student response rate (after rounding) of at least 85 percent 

OR 

The product of the (unrounded) weighted school response rate without replacement and the (unrounded) 
weighted student response rate of at least 75 percent (after rounding to the nearest whole percent). 
Participants in this category appear in the tables and figures in international reports without annotation, 
and are ordered by achievement score. 

Category 2: Met requirements after substitutes. In the case of participants not meeting the category 1 
requirements, but who had a weighted school response rate of at least 50 percent without replacement 
(after rounding to the nearest percent) AND HAD EITHER: 
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A weighted school response rate of at least 85 percent with replacement (after rounding to nearest whole 
percent) AND a weighted student response rate (after rounding) of at least 85 percent. 

OR 

The product of the (unrounded) weighted school response rate with replacement and the (unrounded) 
weighted student response rate of at least 75 percent (after rounding to the nearest whole percent). 

Those participants able to satisfy only the category 2 standard are included in the tables and figures as 
well but are annotated to indicate their response rate status. 

Category 3: Unacceptable sampling response rate even when replacement schools are included. 

Participants that could provide documentation to show that they complied with TIMSS and PIRLS 
sampling procedures and requirements but did not meet the requirements for Category 1 or Category 2 
were be placed in Category 3. Participants in this category appeared in a separate section of the 
achievement tables, below the other participants, in international reports. These countries were presented 
in alphabetical order. 


3.1 TIMSS and PIRLS Participation Rates of U.S. Schools, 
Classrooms, and Students 

The raw numbers on which the various participation rates are based, along with the participation rates 
themselves, are shown in table 3-1 separately for the TIMSS fourth-grade and eighth-grade national 
samples and table 3-2 for the PIRLS national sample. Participation rates for the state samples are shown 
in appendix B. 

To explain how to interpret these participation rates, subsections 3.1.1 through 3.1.3 describe a complete 
interpretation of the TIMSS fourth-grade numbers. The participation rates for TIMSS eighth-grade, 
PIRLS, and all the state samples can be interpreted in the same way. 
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Table 3-1. Number of U.S. schools, classrooms, and students participating in TIMSS, and participation 
rates, by grade: 2011 


Item 

Number 

Grade 4 

Rates 

Grade 8 

Grade 4 

Grade 8 

Unweighted Weighted 

Unweighted Weighted 

Schools 






Sampled 

450 

600 




Excluded and ineligible 

13 

26 




Eligible 

437 

574 




Participating 

347 

499 

79 

79 

87 87 

Substitutes 

22 

2 




Participating (all schools) 

369 

501 

84 

84 

87 87 

Classrooms in participating 






schools 






Total 

1,674 

5,199 




Excluded 

72 

536 




Eligible 

1,602 

4,663 




Sampled 

623 

557 




Participating 

623 

557 

100 

100 

100 100 

Students in participating 






schools 






Sampled 

14,205 

11,864 




Excluded 

839 

398 




Withdrawn 

185 

302 




Eligible 

13,181 

11,164 




Absent 

612 

687 




Assessed 

12,569 

10,477 

95 

95 

94 94 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools since substitute schools do not have an independent probability of selection 
(National Center for Education Statistics 2002). However, the participation rates shown in this table are 
those reported by TIMSS and do include substitute schools in the calculations. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 3-2. Number of U.S. schools, classrooms, and students participating in PIRLS, and participation 
rates, for fourth-grade: 2011 



Number 

Rates 

Unweighted 

Weighted 

Schools 




Sampled 

450 



Excluded and ineligible 

13 



Eligible 

437 



Participating 

349 

80 

80 

Substitutes 

21 



Participating (all schools) 

370 

80 

85 

Classrooms in participating schools 




Total 

1,674 



Excluded 

72 



Eligible 

1,602 



Sampled 

618 



Participating 

618 

100 

100 

Students in participating schools 




Sampled 

14,253 



Excluded 

830 



Withdrawn 

169 



Eligible 

13,254 



Absent 

528 



Assessed 

12,726 

96 

96 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools since substitute schools do not have an independent probability of selection 
(National Center for Education Statistics 2002). However, the participation rates shown in this table are 
those reported by PIRLS and do include substitute schools in the calculations. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 


Interpreting School Participation Rates (TIMSS Fourth Grade Example) 

The fourth-grade school sample consisted of 450 schools and was designed to yield a representative 
school sample for TIMSS and PIRLS, as both assessments were administered within the same schools . 
Thirteen ineligible schools were identified on the basis that they served special student populations or had 
closed or altered their grade makeup since the sampling frame was developed. This left 437 eligible 
schools, of which 347 agreed to participate. The fourth-grade school participation rate before substitution 
was 79 percent (unweighted). The analogous weighted school participation rate was also 79 percent. 
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In addition to the 347 participating schools from the original sample, 22 substitute schools also 
participated for a total of 369 participating schools at the fourth grade in the United States. This gave a 
weighted (and unweighted) school participation rate after substitution of 84 percent. 


Interpreting Classroom Participation Rates (TIMSS Fourth Grade Example) 

In accord with the international requirements, schools agreeing to participate were asked to list their 
fourth-grade mathematics classes as the basis for sampling at the classroom level. Schools appeared to be 
able to identify classes in this way without any problems. A total of 1,674 mathematics classrooms were 
identified as a result. At this time, schools were given the opportunity to identify special classes — classes 
containing all, or a majority of, students with intellectual and/or functional disabilities, or students who 
were non-native -language speakers. While these classes were regarded as eligible, the students as a group 
were treated as excluded since, in the opinion of the school, their disabilities or language capabilities 
would render meaningless their performance on the assessment. A total of 72 classrooms were excluded 
in this way. This left a pool of 1 ,602 eligible classrooms from which the sample was drawn. While the 
students in these excluded classrooms did not figure in the participation rate calculations, they did count 
in the population coverage calculations, and this is reflected in the higher exclusion rate for the U.S. In 
the international report the U.S. is annotated to reflect this fact. 

Classrooms with fewer than 1 5 students were collapsed into pseudoclassrooms prior to sampling so that 
each eligible classroom in a school had at least 20 students. Two classrooms (pseudoclassrooms) were 
selected per school where possible. In schools where there was only one classroom, this classroom was 
selected with certainty. Some 623 classrooms were selected as a result of this process, and all participated 
in TIMSS. Weighted and unweighted classroom participation rates were 100 percent. 

Subsequently, schools were asked to list the students in each of the 623 sampled classrooms at the fourth 
grade, along with the teachers who taught mathematics and science to these students. At this time, schools 
were given the opportunity to identify particular students not suited to take the test because of functional 
and/or intellectual disabilities and/or because they were non-native language speakers (students with 
disabilities or non-native-language speakers who had been mainstreamed; see definitions in section 3.3). 
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Interpreting Student Participation Rates (TIMSS Fourth Grade Example) 


A total of 14,205 fourth-grade students were listed as being in these classrooms. (In mixed-grade 
classrooms only students in the target population were considered.) At the outset, 839 of these were 
excluded because of functional/intellectual disabilities or because they were non-native language 
speakers. Additionally, in the months between the listing of students and the time of the assessment, 185 
students were classified as withdrawn, as they were no longer in the school/classroom at the time of the 
assessment. As a consequence, 13,181 students were considered eligible to take the assessment. On the 
day of the assessment some 612 students were absent, leaving 12,569 students who completed a TIMSS 
2011 assessment booklet. Participation rates are calculated on the number of eligible students (13,181). 
Since 12,569 of the 13,181 eligible students were assessed, the weighted (and unweighted) student 
participation rate was 95 percent. 


Combined Participation Rates 

The combined school, classroom, and student weighted participation rate standard of 75 percent used by 
TIMSS and PIRLS in situations in which it was necessary to recruit substitute schools was met for all 
grade samples in TIMSS and PIRLS 2011. Both the weighted and unweighted product of the separate 
participation rates for TIMSS at grade 4 was 80 percent (82 percent for both TIMSS at grade 8 and 
PIRLS). The application of international guidelines means, however, that U.S. statistics describing fourth- 
grade students in TIMSS and PIRLS are annotated in international reports to indicate that coverage of the 
defined student population was less than the lEA standard of 95 percent and that participation rates were 
met only after substitute schools were included. 


3.2 Participation Rates for Aii Countries 

For comparable fourth- and eighth-grade school, classroom, and student participation rates in other 
nations in TIMSS, see exhibits C-2. through C-9 in appendix C of Martin et. al. (2012) and, for PIRLS 
see exhibits C-2. through C-9 in appendix C of Mullis et. al. (2012). 
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3.3 


Exclusions 


The national defined target population is described in Sample Design in TIMSS and PIRLS (Joncas & 
Foy, 2013). All schools and students excluded from this population are referred to as the “excluded 
population.” Exclusions could occur at the school level, with entire schools being excluded, or within 
schools, with specific students or entire classrooms excluded. TIMSS 2011 and PIRES 2011 did not 
provide accommodations for students with disabilities or students who were unable to read or speak the 
language of the test. 


School Exclusions 


Countries could exclude schools that 


■ were geographically inaccessible; 

■ were of extremely small size; 

■ offered a curriculum or school structure radically different from the mainstream 
educational system; or 

■ provided instruction only to students in the excluded categories defined under “within- 
school exclusions,” such as schools for the blind. 


Within-School Exclusions 

Countries were asked to adapt the following international within-school exclusion rules to define 
excluded students: 

Students with intellectual disabilities. Students who, in the professional opinion of the school principal 
or other qualified staff members, were considered to be intellectually disabled or who had been tested 
psychologically as such. This included students who were emotionally or mentally unable to follow even 
the general instructions of the test. Students were not to be excluded solely because of poor academic 
performance or normal disciplinary problems. 

Students with functional disabilities. Students who were permanently physically disabled in such a way 
that they could not perform in the TIMSS or PIRLS testing situation. Functionally disabled students who 
were able to respond were included in the testing. 
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Non-native-language speakers. Students who were unable to read or speak the language(s) of the test 
and were unable to overcome the language barrier of the test. Typically, a student who had received less 
than 1 year of instruction in the language(s) of the test was excluded. 


Exclusions in the U.S. National Samples 

As noted earlier, schools were given the opportunity to exclude any special classes among the total 
number of classes in the fourth or eighth grade. These classes were made up largely of students with 
functional or intellectual disabilities or students who were non-native-language speakers, as defined 
above. Classes identified in this way were excluded from the class sampling procedure. Subsequently, 
schools were given the opportunity to exclude students from the sampled classes — essentially, students 
with functional or intellectual disabilities, or non-native-language speaking students in the United States 
who had been mainstreamed. 

These procedures resulted in a (weighted) student exclusion rate of 7.0 percent in the fourth grade for 
TIMSS, 7.3 percent for PIRLS, and 7.2 percent for TIMSS in the eighth grade, based on the combination 
of whole-class and within-class exclusions. lEA standards define this degree of coverage of the national 
target population (93 percent for TIMSS at both fourth and eighth grade and PIRLS) as acceptable though 
falling below the desired range of 95 percent or better. The tabulations shown in the international reports 
show the United States annotated to indicate this fact. 

Exclusion information for the state samples can be found on appendix B. 


3.4 Nonresponse Bias Analysis 

The National Center for Education Statistics (NCES) standards for assessment surveys stipulate that a 
nonresponse bias analysis is required at any stage of data collection with a weighted unit response rate of 
less than 85 percent (without substitution) at both grades. Because the U.S. TIMSS and PIRLS 2011 
weighted school response rates at grade 4 are below 85 percent, NCES required an investigation into the 
potential magnitude of nonresponse bias at the school level in the U.S. samples, which is the focus of this 
section. Neither the U.S. TIMSS grade 8 nor any of the benchmarking states required a nonresponse bias 
analysis be performed because their weighted school and student participation rates were above 85 
percent. 
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Methodology 


To measure the potential nonresponse bias at the school level, the characteristics of participating schools 
were compared to those of the total eligible sample of schools. The alternative of comparing participants 
to nonparticipants, while resulting in the same tests of significance, makes it more difficult to judge the 
potential for bias. This analysis is similar to other NCES nonresponse bias studies on the 2003 TIMSS 
(Ferraro & Van de Kerckhove, 2006). 

The analysis for each sample (TIMSS Grade 4 and PIRLS Grade 4) was conducted in three parts as 
follows: 

Analysis of original respondent sample: The distribution for TIMSS of the responding (participating) 
original school sample (N = 347) was compared with that of the total eligible original school sample (N = 
437). The distribution for PIRLS of the responding (participating) original school sample (N = 349) was 
compared with that of the total eligible original school sample (N = 437). The original sample is the 
sample before substitution. In each sample, schools were weighted by their school base weights that did 
not include a nonresponse adjustment factor. The base weight for each original school was the reciprocal 
of its selection probability. 

Analysis of respondent sample with substitutes (final sample): The distribution for TIMSS of the 
responding (participating) sample with substitutes (N = 369), was compared to the total eligible final 
sample (N = 437). The distribution for PIRLS of the responding (participating) sample with substitutes (N 
= 370), was compared to the total eligible final sample (N = 437). The final sample is the sample after 
substitution. Again, school base weights were used for both the eligible sample and the participating 
schools. The base weight for each substitute school was set to the base weight of the original school that it 
replaced. 

Analysis of the nonresponse adjusted sample with substitutes: As done in the second analysis, the 
same sets of schools were compared (i.e., N = 369 vs. N = 437 for TIMSS, N = 370 vs. N = 437 for 
PIRLS), but, this time, when analyzing the responding schools alone, school nonresponse adjustments 
were applied to the weights. The international weighting procedures created a nonresponse adjustment 
class'^ for each explicit stratum. For U.S. TIMSS grade 4, or “TIMSS-4,” and U.S. PIRLS grade 4, or 


general, nonresponse adjustment classes are formed based on characteristics related to response rates or to values of survey estimates where 
respondents and nonrespondents are similar within each class. The nonresponse adjustment is applied within each of these classes. 
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“PlRLS-4,” 12 explicit strata were formed by poverty level, school control, and Census region, thus 
forming 12 adjustment classes. 

The first analysis indicates the potential for nonresponse bias that was introduced through school 
nonresponse. The second analysis suggests the remaining potential for nonresponse bias after the 
mitigating effects of substitution have been accounted for. The third analysis indicates the potential for 
bias after accounting for the mitigating effects of both substitution and nonresponse weight adjustments. 
Both the second and third analyses, however, may provide an overly optimistic scenario because even 
though substitution and nonresponse adjustments may correct somewhat for deficiencies in the few 
characteristics examined here, there is no guarantee that they are equally as effective for other 
characteristics, and in particular for student achievement. 

To compare participants and the total eligible sample, the sample of schools was matched to the sample 
frame to compare as many characteristics as possible that might provide information about the presence 
of nonresponse bias. Since the analyses involve both participating and nonparticipating schools, they are 
based, out of necessity, on data from the sampling frame as TIMSS and PIRLS data are not available for 
nonparticipating schools. Comparing frame characteristics for participants and the total eligible sample is 
not an ideal measure of nonresponse bias if the characteristics are unrelated or weakly related to more 
substantive items in the survey; however, this is often the only approach available. 

Frame characteristics for public schools were taken from the 2007-08 Common Core of Data (CCD) and, 
for private schools, from the 2007-08 Private School Survey (PSS). 

The following categorical variables were available in the sampling frame for all schools: 

School control. Indicates whether the school is under public control (operated by publicly elected or 
appointed officials) or private control (operated by privately elected or appointed officials and derives its 
major source of funds from private sources); 

Community type. The location of a school relative to populous areas, i.e., city, suburban, town, rural; 
Region. Census region (see section 3.5 for state listing); and 
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Poverty level/'* For public schools, a high poverty school is defined as one in which 50 percent or more 
of the students are eligible for the free or reduced-price lunch (FRPL) program, and a low poverty school 
is defined as one in which less than 50 percent are eligible; all private schools are treated as low-poverty 
schools. 

The following continuous variables were available in the sampling frame for all schools: 

■ number of grade 4 students enrolled; 

■ total number of students; and 

■ percentage of students in five race/ethnicity categories (White, non-Flispanic; Black, 
non-Flispanic; Flispanic; Asian or Pacific Islander; and American Indian or Alaska 
Native).*^ 

An additional continuous variable, the percentage of students eligible to participate in the FRPL, was 
available only for public schools. “Poverty level” variable mentioned among the categorical variable is 
the recoded version of this continuous variable. 

For categorical variables, the distribution of frame characteristics for participants was compared with the 
distribution for all eligible schools. The hypothesis of independence between the characteristic and 
participation status was tested using a Rao-Scott modified chi-square statistic at the 5 percent level (Rao 
& Thomas, 2003). For continuous variables, summary means were calculated, and the difference between 
means was tested using a t test. The statistical significance of differences between participants and the 
total eligible sample is identical to that which would result from comparing participants and 
nonparticipants, since all significance tests account for the fact that the participants are a subset of the full 
sample. The bias and relative bias are also given in each table. The bias is the difference between the 
respective estimates for the participants and the eligible sample. The relative bias is calculated as the bias 
divided by the estimate from the eligible sample. The relative bias is a measure of the size of the bias 
compared to the eligible sample estimate. 

In addition to these tests, logistic regression models were used to provide a multivariate analysis in which 
the conditional independence of these school characteristics as predictors of participation was examined. 

It may be that only one or two variables are actually related to participation status. Flowever, if these 

^^The sample frame did not contain a direct measure of poverty. No free or reduced-price lunch (FRPL) program data were available for private 
schools. 

^^Black includes African American and Hispanic includes Latino. Racial categories exclude Hispanic origin. Asian and Pacific Islander are 
combined into a single category due to the fact that that the frame was developed using the previous NCES guidelines for combining Asian and 
Pacific Islanders, while the U.S. national report was developed under new guidelines that call for reporting out Asian separately when possible. 
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variables are also related to the other variables examined in the analyses, then other variables, which are 
not related to participation status, will appear as significant in simple bivariate tables. Dummy variables 
were created for each component of the categorical variables so that each component was included 
separately. The last component of each categorical variable is always the reference category and is not 
included in the model explicitly. The p value of a dummy variable indicates whether there is a significant 
difference at the 5 percent level from the effect of the (omitted) reference category. It is not possible to 
include all the frame characteristics in a single model because the five race/ethnicity variables are linearly 
dependent (i.e., they sum up to 100 percent for every school). Therefore, two models were used. In the 
first model, four race/ethnicities (Black, non-Hispanic; Hispanic; Asian or Pacific Islander; and American 
Indian or Alaska Native) were included in the model with percentage White, non-Hispanic as the 
reference category. In addition, an F test was used to determine whether the parameter estimates of these 
four characteristics were simultaneously equal to zero. In the second model, the summed percentage of 
the four race/ethnicities (Black, non-Hispanic; Hispanic; Asian or Pacific Islander; and American Indian 
or Alaska Native) replaced the four race/ethnicity variables with percentage White, non-Hispanic again as 
the reference category. All other frame characteristics were included in both models. 

The logistic regression was performed using WesVar (Westat, 2007) 

( http://www.westat.com/expertise/information systems/WesVar/index.cfm) with replicate weights to 
properly account for the complex sample design. The paired jackknife replication method was used to 
create the replicate weights (Brick, Morganstein, & Valliant, 2000). 


Results for Original Respondent Sample — TIMSS Fourth Grade 

This section presents the results of the nonresponse bias analysis, based exclusively on the original 
sample of 437 eligible U.S. schools for TIMSS-4 (section 3.4.6 presents the results for PIRLS). The 
distribution of the responding original school sample was compared with that of the total eligible original 
school sample using base weights in each case. All original schools in the sample that declined to 
participate in the survey were treated as nonparticipants regardless of whether they were substituted by a 
substitute school. The weighted and unweighted response rates were 79 percent, with 347 out of 437 
eligible schools participating. See table 3-1 for details on the TIMSS-4 school participation rates. 

Categorical variables (TIMSS-4). The distribution of participating and eligible schools in the U.S. 
TlMSS-4 sample by the four characteristics is shown in table 3-3. Based on these analyses, the chi-square 
statistics for school control and community type are significant and suggests that there is evidence of 
relationships with participating in the assessment. In particular, public schools were overrepresented 
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among participating schools (96.3 vs. 91.3 percent, respectively), and private schools were 
underrepresented among participating schools (3.7 vs. 8.7 percent, respectively). Similarly, schools in 
cities were underrepresented among participating schools relative to eligible schools (26.8 vs. 30.2 
percent, respectively), while schools in rural areas were overrepresented among participating schools 
(24.2 vs. 22.6 percent, respectively). There are no statistically significant relationships between 
participation status and any of the other characteristics shown in table 3-3. 


Table 3-3. Percentage distribution of eligible and participating schools in the U.S. TIMSS fourth-grade 
original sample, by selected categorical variables: 2011 


School characteristic 

Sample schools 
Eligible Participating 

(percent) (percent) 

(N = 437) (N = 347) 

Bias 

Relative 

bias 

Chi- 
square 
p value 

School control 
Public 

91.3 

96.3 

4.92 

0.054 

0.000 

Private 

8.7 

3.7 

-4.92 

-0.568 


Community type 
City 

30.2 

26.8 

-3.44 

-0.114 

0.010 

Suburban 

35.5 

36.3 

0.80 

0.023 


Town 

11.7 

12.7 

0.99 

0.085 


Rural 

22.6 

24.2 

1.65 

0.073 


Region 

Northeast 

16.4 

15.4 

-1.00 

-0.061 

0.237 

Midwest 

21.2 

20.5 

-0.70 

-0.033 


South 

38.4 

40.7 

2.31 

0.060 


West 

24.0 

23.4 

-0.60 

-0.025 


Poverty level 
High 

39.9 

40.1 

0.18 

0.004 

0.870 

Eow 

60.1 

59.9 

-0.18 

-0.003 



NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/defmitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Eligible schools contained at 
least one fourth-grade class. Participating schools agreed to have their students assessed. The bias is the 
difference between the respective estimates for the participants and the eligible sample. The relative bias 
is calculated as the bias divided by the estimate from the eligible sample. Schools were weighted by their 
school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


47 


Continuous variables (TIMSS-4), Summary means for each continuous variable for participating and 
eligible schools are shown in tables 3-4 through 3-6. Twenty-four of the 399 public schools had a missing 
value for the free or reduced-price lunch variable; these schools were excluded from the analysis. 


Participating schools had a higher mean fourth-grade enrollment than the eligible sample (99.3 vs. 94.0, 
respectively; table 3-4). Participating schools had a lower mean percentage than the eligible sample of 
Black, non-Hispanic students (13.3 vs. 15.3 percent, respectively; table 3-5). Participating schools had a 
higher mean percentage than the eligible sample Hispanic students (25.1 vs. 23.5 percent, respectively; 
table 3-5) and American Indian or Alaska Native students (0.9 vs. 0.8 percent, respectively; table 3-5). 
There were no statistically significant differences detected between the participating and eligible public 
schools for the free or reduced-price lunch (table 3-6). However, this must be interpreted with caution 
because this variable was missing for 24 public schools (see note in table 3-6). 


Table 3-4. Mean enrollment of eligible and participating schools in the U.S. TIMSS fourth-grade 
original sample: 2011 



Sample schools 




Student enrollment 

Eligible 
(mean) 
(N = 437) 

Participating 
(mean) 
(N = 347) 

Bias 

Relative 

bias 

t test 
p value 

Total school 

561.3 

570.1 

8.74 

0.016 


Fourth grade 

94.0 

99.3 

5.31 

0.056 



NOTE: Eligible schools contained at least one fourth-grade class. Participating schools agreed to have 
their students assessed. The bias is the difference between the respective estimates for the participants and 
the eligible sample. The relative bias is calculated as the bias divided by the estimate from the eligible 
sample. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 3-5. Mean percentage of students in eligible and participating schools in the U.S. TIMSS fourth- 
grade original sample, by race/ethnicity: 2011 


Sample schools 


Race/ethnicity 

Eligible 
(percent) 
(N = 437) 

Participating 
(percent) 
(N = 347) 

Bias 

Relative 

bias 

t test 
p value 

White, non-Hispanic 

55.9 

56.2 

0.28 

0.005 

0.761 

Black, non-Hispanic 

15.3 

13.3 

-2.00 

-0.131 

0.015 

Hispanic 

23.5 

25.1 

1.66 

0.071 

0.011 

Asian or Pacific Islander 

4.5 

4.5 

-0.04 

-0.010 

0.766 

American Indian or Alaska 
Native 

0.8 

0.9 

0.11 

0.133 

0.033 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Eligible schools contained at least one fourth-grade class. Participating schools agreed to 
have their students assessed. The bias is the difference between the respective estimates for the 
participants and the eligible sample. The relative bias is calculated as the bias divided by the estimate 
from the eligible sample. Schools were weighted by their school base weights that did not include a 
nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 3-6. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. TIMSS fourth-grade original sample: 2011 



Sample schools 



Students 

Eligible 
(percent) 
(N = 375) 

Participating 
(percent) 
(N = 313) 

Relative 
Bias bias 

t test 
p value 

Percentage of students eligible for 
free or reduced-price lunch 

48.1 

47.2 

-0.90 -0.019! 

0.170! 


! Interpret data with caution. 

NOTE: Information on percentage of students eligible for free or reduced-price lunch is missing for 24 of 
the 399 public schools in the original sample and 21 of the 334 public schools that participated. Eligible 
schools contained at least one fourth-grade class. Participating schools agreed to have their students 
assessed. The bias is the difference between the respective estimates for the participants and the eligible 
sample. The relative bias is calculated as the bias divided by the estimate from the eligible sample. 

Schools were weighted by their school base weights that did not include a nonresponse adjustment factor. 
SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Logistic regression model (TIMSS-4), To examine the joint relationship of various characteristics to 
school nonresponse, the analysis utilized a logistic regression model with participation status as the binary 
dependent variable and frame characteristics as predictor variables. Public and private schools were 
modeled together using the variables available for all schools. 

Standard errors and tests of hypotheses for the full model parameter estimates are presented in tables 3 -7a 
(with four race/ethnicity variables) and 3-7b (with summed race/ethnicity percentage). Private schools, 
South region, total school enrollment, fourth-grade enrollment, the percentage of Hispanic students, and 
the percentage of American Indian or Alaska Native students are significant predictors of school 
participation in table 3 -7a. The negative parameter estimates indicate that, relative to public schools, 
private schools were somewhat underrepresented among the participating schools, and the total 
enrollment in participating schools was slightly smaller than in all eligible schools, i.e., the smaller the 
total enrollment, the more likely a school was to participate. The positive parameter estimates indicate 
that, relative to schools in the West region, schools in the South region were somewhat overrepresented 
among the participating schools, and the fourth-grade enrollment, percentage of Hispanic students, and 
percentage of American Indian or Alaska Native students in participating schools was larger than in all 
eligible schools. The F test statistic to determine whether the race/ethnicity characteristics are 
simultaneously equal to 0 was 2.86 with a p value of 0.0023, which indicates a significant relationship 
detected with participation. 

Private schools, total school enrollment, and fourth-grade enrollment again are significant predictors of 
school participation in table 3-7b. This model also shows that the summed race/ethnicity percentage is not 
significantly related to participation. 
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Table 3 -7a. Logistic regression model parameter estimates (with four race/ethnicity variables) using the 
U.S. TIMSS fourth-grade original sample (N = 437): 2011 


Parameter 

Parameter 

estimate 

Standard 

error 

t test for Hq: 
parameter = 0 

p value 

Intercept 

1.942 

0.3693 

5.2575 

0.0000 

City 

-0.436 

0.4149 

-1.0521 

0.2961 

Suburban 

-0.116 

0.3450 

-0.3351 

0.7385 

Town 

0.076 

0.5896 

0.1287 

0.8979 

Private school 

-2.302 

0.5561 

-4.1393 

0.0001 

High poverty 

-0.643 

0.3298 

-1.9491 

0.0550 

Northeast 

0.416 

0.4341 

0.9573 

0.3415 

Midwest 

0.421 

0.3719 

1.1318 

0.2613 

South 

1.099 

0.4354 

2.5237 

0.0137 

Total school enrollment 

-0.002 

0.0007 

-2.6538 

0.0097 

Fourth-grade enrollment 

0.012 

0.0056 

2.1574 

0.0342 

Black, non-Hispanic 

-0.011 

0.0060 

-1.8377 

0.0701 

Hispanic 

0.015 

0.0072 

2.0644 

0.0424 

Asian or Pacific Islander 

-0.002 

0.0165 

-0.1442 

0.8857 

American Indian or Alaska Native 

0.251 

0.1063 

2.3621 

0.0208 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/definitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Black includes African 
American, and Hispanic includes Fatino. Racial categories exclude Hispanic origin. Schools were 
weighted by their school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 3-7b. Logistic regression model parameter estimates (with summed race/ethnicity percentage) 
using the U.S. TIMSS fourth-grade original sample (N=437): 2011 


Parameter 

Parameter 

estimate 

Standard error 

t test for Ho: 
parameter = 0 

p value 

Intercept 

2.300 

0.3801 

6.0513 

0.0000 

City 

-0.586 

0.4289 

-1.3653 

0.1763 

Suburban 

-0.179 

0.4145 

-0.4328 

0.6664 

Town 

0.134 

0.5614 

0.2378 

0.8127 

Private school 

-1.934 

0.5480 

-3.5289 

0.0007 

High poverty 

-0.509 

0.3546 

-1.4364 

0.1550 

Northeast 

-0.189 

0.4172 

-0.4518 

0.6527 

Midwest 

-0.113 

0.3572 

-0.3164 

0.7526 

South 

0.382 

0.3218 

1.1867 

0.2391 

Total school enrollment 

-0.002 

0.0007 

-2.2225 

0.0293 

Fourth-grade enrollment 

0.014 

0.0065 

2.1496 

0.0348 

Summed race/ethnicity percentage 

0.002 

0.0059 

0.3215 

0.7487 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Summed race/ethnicity 
percentage includes Black, non-Hispanic; Hispanic; Asian or Pacific Islander; and American Indian or 
Alaska Native. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Results for Respondent Sample With Substitutes (Final Sample) — TIMSS 
Fourth Grade 

This section presents the nonresponse bias analysis based on the final sample of 437 eligible schools for 
TlMSS-4. The distribution of the responding sample, including participating substitutes, was compared to 
the total eligible final sample. School base weights were used for both the eligible sample and the 
participating schools. Only eligible original schools that refused and were not successfully replaced by a 
substitute were treated as nonparticipants. All other eligible original sample schools were treated as 
participating. Through the use of substitutes, the weighted and unweighted school response rates for 
TIMSS-4 were 84 percent, with 369 out of 437 schools participating. 

Categorical variables (TIMSS-4). The distribution of participating and eligible schools by the four 
characteristics is shown in table 3-8. Only school control was found to be statistically significant among 
the categorical variables. In particular, public schools were overrepresented among participating schools 
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(96.2 vs. 91.3 percent, respectively), and private schools were underrepresented among participating 
schools (3.8 vs. 8.7 percent, respectively). These differences were slightly smaller than that shown in 
table 3-3, in which only the original sample was considered. Thus while there is no evidence that the use 
of substitute schools eliminated the potential for bias, as indicated by this variable, it certainly has also 
not added to it. There are no statistically significant relationships detected between participation status 
and the other characteristics shown in table 3-8. 


Table 3-8. Percentage distribution of eligible and participating schools in the U.S. TIMSS fourth-grade 
final sample, by selected categorical variables: 2011 


School characteristic 

Sample schools 
Eligible Participating 
(percent) (percent) 

(N = 437) (N = 369) 

Bias 

Relative 

bias 

Chi-square 
p value 

School control 





0.000 

Public 

91.3 

96.2 

4.88 

0.053 


Private 

8.7 

3.8 

-4.88 

-0.564 


Community type 





0.155 

City 

30.2 

28.2 

-2.06 

-0.068 


Suburban 

35.5 

36.0 

0.54 

0.015 


Town 

11.7 

12.2 

0.50 

0.043 


Rural 

22.6 

23.6 

1.02 

0.045 


Region 





0.630 

Northeast 

16.4 

15.5 

-0.82 

-0.050 


Midwest 

21.2 

20.9 

-0.29 

-0.014 


South 

38.4 

39.1 

0.70 

0.018 


West 

24.0 

24.5 

0.41 

0.017 


Poverty level 





0.271 

High 

39.9 

41.0 

1.04 

0.026 


Eow 

60.1 

59.0 

-1.04 

-0.017 



NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the ifee or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Eligible schools contained at 
least one fourth-grade class. Participating schools agreed to have their students assessed. The bias is the 
difference between the respective estimates for the participants and the eligible sample. The relative bias 
is calculated as the bias divided by the estimate from the eligible sample. Schools were weighted by their 
school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Continuous variables (TIMSS-4), Summary means for each continuous variable for participating and 
eligible schools are shown in tables 3-9 through 3-11. Twenty-four of the 399 public schools had a 
missing value for the free or reduced-price lunch variable; these schools were excluded from the analysis. 

Participating schools had a higher mean fourth-grade enrollment than the eligible sample (99.1 vs. 94.0, 
respectively; table 3-9). Participating schools had a lower mean percentage than the eligible sample of 
Black, non-Hispanic students (13.4 vs. 15.2 percent, respectively; table 3-10). Participating schools had a 
higher mean percentage than the eligible sample Hispanic students (25.0 vs. 23.1 percent, respectively; 
table 3-10) and American Indian or Alaska Native students (1.0 vs. 0.9 percent, respectively; table 3-10). 
The difference in the fourth-grade enrollment was slightly smaller than that shown in table 3-4, in which 
only the original sample was considered. The differences in the percentage of students who are Black, 
non-Hispanic and Hispanic are somewhat smaller than that shown in table 3-5, in which only the original 
sample was considered. Thus, as in the case with school control, while there is no evidence that the use of 
substitute schools substantially reduced the potential for bias, as indicated by these variables, it has also 
not added to it. 


There was no statistically significant difference detected between the participating and eligible public 
schools for free or reduced-price lunch (table 3-11). However, this must be interpreted with caution 
because the variable is missing for 24 schools. 


Table 3-9. Mean enrollment of eligible and participating schools in the U.S. TIMSS fourth-grade final 
sample: 2011 



Sample schools 




Student enrollment 

Eligible 
(mean) 
(N = 437) 

Participating 
(mean) 
(N = 369) 

Bias 

Relative 

bias 

t test 
p value 

Total school 

562.2 

572.3 

10.05 



Fourth grade 

94.0 

99.1 

5.08 




NOTE: Eligible schools contained at least one fourth-grade class. Participating schools agreed to have 
their students assessed. The bias is the difference between the respective estimates for the participants and 
the eligible sample. The relative bias is calculated as the bias divided by the estimate from the eligible 
sample. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 3-10. Mean percentage of students in eligible and participating schools in the U.S. TIMSS fourth- 
grade final sample, by race/ethnicity: 2011 


Sample schools 


Race/ethnicity 

Eligible 

(percent) 

(N=437) 

Participating 
(percent) 
(N = 369) 

Bias 

Relative 

bias 

t test 
p value 

White, non-Hispanic 

56.2 

55.9 

-0.32 

-0.01 

0.690 

Black, non-Hispanic 

15.2 

13.4 

-1.77 

-0.12 

0.024 

Hispanic 

23.1 

25.0 

1.89 

0.08 

0.001 

Asian or Pacific Islander 

4.7 

4.8 

0.09 

0.02 

0.426 

American Indian or Alaska Native 

0.9 

1.0 

0.10 

0.12 

0.008 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Eligible schools contained at least one fourth-grade class. Participating schools agreed to 
have their students assessed. The bias is the difference between the respective estimates for the 
participants and the eligible sample. The relative bias is calculated as the bias divided by the estimate 
from the eligible sample. Schools were weighted by their school base weights that did not include a 
nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Table 3-11. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. TIMSS fourth-grade final sample: 2011 



Sample schools 



Students 

Eligible 
(percent) 
(N = 375) 

Participating 
(percent) 
(N = 333) 

Relative 
Bias bias 

t test 
p value 

Percentage of students eligible for 
free or reduced-price lunch 

47.9 

47.2 

-0.70 -0.015 

0.217 


NOTE: Information on percentage of students eligible for free or reduced-price lunch is missing for 24 of 
the 399 public schools in the final sample and 22 of the 355 public schools that participated. Eligible 
schools contained at least one fourth-grade class. Participating schools agreed to have their students 
assessed. The bias is the difference between the respective estimates for the participants and the eligible 
sample. The relative bias is calculated as the bias divided by the estimate from the eligible sample. 

Schools were weighted by their school base weights that did not include a nonresponse adjustment factor. 
SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Logistic regression model (TIMSS-4), To examine the joint relationship of various characteristics to 
school nonresponse, the analysis utilized a logistic regression model with participation status as the binary 
dependent variable and frame characteristics as predictor variables. Six schools were excluded from the 
analysis due to missing information for race/ethnicity and total school enrollment. 
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Private schools, total school enrollment, and the percentage of American Indian or Alaska Native students 
remained significant predictors of school participation in table 3-1 2a. The negative parameter estimates 
indicate that, relative to public schools, private schools were somewhat underrepresented among the 
participating schools, and the total enrollment in participating schools was smaller than in all eligible 
schools, i.e., the smaller the total enrollment, the more likely a school was to participate. The positive 
parameter estimates indicate that, the percentage of American Indian or Alaska Native students in 
participating schools was lower than in all eligible schools. The A test statistic to determine whether the 
race/ethnicity characteristics are simultaneously equal to 0 was 2.66 with a p value of 0.0042, which 
indicates a significant relationship detected with participation. 


Private schools, total school enrollment, and fourth-grade enrollment were also significant predictors of 
school participation in table 3- 12b. This model also shows that the summed race/ethnicity percentage is 
not significantly related to participation. 


Table 3- 12a. Logistic regression model parameter estimates (with four race/ethnicity variables) in the 
U.S. TIMSS fourth-grade final sample (N = 437): 2011 


Parameter 

Parameter 

estimate 

Standard 

error 

t test for Ho: 
parameter = 0 

p value 

Intercept 

2.017 

0.6166 

3.2707 

0.0016 

City 

-0.293 

0.5195 

-0.5646 

0.5740 

Suburban 

-0.132 

0.4447 

-0.2971 

0.7672 

Town 

-0.128 

0.6554 

-0.1959 

0.8452 

Private school 

-2.375 

0.6501 

-3.6535 

0.0005 

High poverty 

-0.429 

0.4614 

-0.9292 

0.3558 

Northeast 

0.348 

0.5340 

0.6516 

0.5167 

Midwest 

0.509 

0.5740 

0.8860 

0.3784 

South 

0.757 

0.5049 

1.4985 

0.1382 

Total school enrollment 

-0.002 

0.0011 

-2.2424 

0.0279 

Fourth-grade enrollment 

0.020 

0.0108 

1.8537 

0.0677 

Black, non-Hispanic 

-0.013 

0.0067 

-1.9456 

0.0554 

Hispanic 

0.018 

0.0112 

1.6252 

0.1083 

Asian or Pacific Islander 

0.001 

0.0155 

0.0773 

0.9386 

American Indian or Alaska Native 

0.290 

0.1337 

2.1705 

0.0331 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the ifee or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Black includes African 
American, and Hispanic includes Fatino. Racial categories exclude Hispanic origin. Schools were 
weighted by their school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


56 


Table 3-12b. Logistic regression model parameter estimates (with summed race/ethnicity percentage) in 
the U.S. TIMSS fourth-grade final sample (N = 437): 2011 


Parameter 

Parameter 

estimate 

Standard t test for Hq: 

error parameter = 0 

p value 

Intercept 

2.522 

0.6195 

4.0702 

0.0001 

City 

-0.496 

0.5331 

-0.9312 

0.3547 

Suburban 

-0.205 

0.4553 

-0.4508 

0.6535 

Town 

-0.093 

0.6003 

-0.1551 

0.8771 

Private school 

-1.864 

0.6465 

-2.8824 

0.0051 

High poverty 

-0.376 

0.4544 

-0.8266 

0.4111 

Northeast 

-0.430 

0.5191 

-0.8282 

0.4102 

Midwest 

-0.226 

0.4590 

-0.4921 

0.6241 

South 

-0.149 

0.3795 

-0.3916 

0.6965 

Total school enrollment 

-0.002 

0.0011 

-2.2352 

0.0284 

Fourth-grade enrollment 

0.024 

0.0113 

2.1027 

0.0389 

Summed race/ethnicity percentage 

0.002 

0.0073 

0.2773 

0.7823 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the Ifee or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Summed race/ethnicity 
percentage includes Black, non-Hispanic; Hispanic; Asian or Pacific Islander; and American Indian or 
Alaska Native. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Results for Nonresponse-Adjusted Sample With Substitutes (Nonresponse- 
Adjusted Sample) — TIMSS Fourth Grade 

This section presents the nonresponse bias analysis based on the final sample of 437 eligible schools for 
TlMSS-4. The distribution of the responding sample, including participating substitutes, was compared to 
the total eligible final sample, just like the previous section. However, in this section, school base weights 
were used for the eligible sample of schools, whereas nonresponse-adjusted weights were used for the 
participating schools. Only eligible original schools that refused and were not successfully replaced by a 
substitute were treated as nonparticipants. All other eligible original sample schools were treated as 
participating. 

Categorical variables (TIMSS-4). The distribution of participating and eligible schools by the four 
characteristics is shown in table 3-13. Only region was found to be statistically significant among the 
categorical variables. In particular, schools in the South and West regions were underrepresented among 
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participating schools relative to eligible schools (37.8 vs. 38.4 and 23.5 vs. 24.0 percent, respectively), 
while schools in the Midwest region were overrepresented among participating schools relative to eligible 
schools (22.4 vs. 21.2 percent, respectively). 


Table 3-13. Percentage distribution of eligible and participating schools in the U.S. TIMSS fourth-grade 
nonresponse-adjusted sample, by selected categorical variables: 2011 


School characteristic 

Sample schools 
Eligible Participating 

(percent) (percent) 

(N = 437) (N = 369) 

Bias 

Relative Chi-square 
bias p value 

School control 





0.995 

Public 

91.3 

91.3 

0.00 

0.000 


Private 

8.7 

8.7 

0.00 

0.000 


Community type 





0.468 

City 

30.2 

29.1 

-1.09 

-0.036 


Suburban 

35.5 

35.9 

0.36 

0.010 


Town 

11.7 

12.3 

0.63 

0.054 


Rural 

22.6 

22.7 

0.10 

0.004 


Region 





0.009 

Northeast 

16.4 

16.4 

0.01 

0.000 


Midwest 

21.2 

22.4 

1.15 

0.054 


South 

38.4 

37.8 

-0.56 

-0.015 


West 

24.0 

23.5 

-0.60 

-0.025 


Poverty level 





0.863 

High 

39.9 

39.9 

-0.06 

-0.002 


Low 

60.1 

60.1 

0.06 

0.001 



NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Eligible schools contained at 
least one fourth-grade class. Participating schools agreed to have their students assessed. The bias is the 
difference between the respective estimates for the participants and the eligible sample. The relative bias 
is calculated as the bias divided by the estimate from the eligible sample. Schools were weighted by their 
school nonresponse adjusted weight. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Continuous variables (TIMSS-4). Summary means for each continuous variable for participating and 
eligible schools are shown in tables 3-14 through 3-16. Twenty-four of the 399 public schools had a 
missing value for the free or reduced-price lunch variable; these schools were excluded from the analysis. 
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There were no statistically significant enrollment differences detected between participating and eligible 
schools (table 3-14). The only remaining significant difference for race/ethnicity was for Black, non- 
Hispanic students where participating schools had a lower mean percentage than the eligible sample of 
(13.0 vs. 15.2 percent, respectively; table 3-15). The difference in the percentage of students who are 
Black, non-Hispanic was somewhat larger than that shown in table 3-10, in which only the original 
sample was considered. 


There was no statistically significant difference detected between the participating and eligible public 
schools for free or reduced-price lunch (table 3-16). However, this must be interpreted with caution 
because the variable is missing for 24 schools. 


Table 3-14. Mean enrollment of eligible and participating schools in the U.S. TIMSS fourth-grade 
nonresponse-adjusted sample: 2011 



Sample schools 




Student enrollment 

Eligible 
(mean) 
(N = 437) 

Participating 
(mean) 
(N = 369) 

Bias 

Relative 

bias 

I test 
p value 

Total school 

562.2 

558.0 

-4.29 


0.654 

Fourth grade 

94.0 

95.4 

1.32 


0.349 


NOTE: Eligible schools contained at least one fourth-grade class. Participating schools agreed to have 
their students assessed. The bias is the difference between the respective estimates for the participants and 
the eligible sample. The relative bias is calculated as the bias divided by the estimate from the eligible 
sample. Schools were weighted by their school nonresponse-adjusted weight. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table 3-15. Mean percentage of students in eligible and participating schools in the U.S. TIMSS fourth- 
grade nonresponse-adjusted sample, by race/ethnicity: 2011 


Sample schools 


Race/ethnicity 

Eligible 
(percent) 
(N = 437) 

Participating 
(percent) 
(N = 369) 

Bias 

Relative 

bias 

t test 
p value 

White, non-Hispanic 

56.2 

56.3 

0.03 

0.00 

0.974 

Black, non-Hispanic 

15.2 

13.0 

-2.20 

-0.14 

0.037 

Hispanic 

23.1 

24.9 

1.86 

0.08 

0.097 

Asian or Pacific Islander 

4.7 

4.7 

0.01 

0.00 

0.980 

American Indian or Alaska Native 

0.9 

1.2 

0.29 

0.32 

0.156 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Eligible schools contained at least one fourth-grade class. Participating schools agreed to 
have their students assessed. The bias is the difference between the respective estimates for the 
participants and the eligible sample. The relative bias is calculated as the bias divided by the estimate 
from the eligible sample. Schools were weighted by their school nonresponse adjusted weight. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 

Table 3-16. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 

participating public schools in the U.S. TIMSS fourth-grade nonresponse-adjusted sample: 
2011 



Sample schools 




Students 

Eligible 
(percent) 
(N = 375) 

Participating 
(percent) 
(N = 333) 

Bias 

Relative 

bias 

t test 
p value 

Percentage of students eligible for 
free or reduced-price lunch 

47.9 

47.6 

-0.30 

-0.006 

0.742 


NOTE: Information on percentage of students eligible for free or reduced-price lunch is missing for 24 of 
the 399 public schools in the final sample and 22 of the 355 public schools that participated. Eligible 
schools contained at least one fourth-grade class. Participating schools agreed to have their students 
assessed. The bias is the difference between the respective estimates for the participants and the eligible 
sample. The relative bias is calculated as the bias divided by the estimate from the eligible sample. 

Schools were weighted by their school nonresponse adjusted weight. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Summary — TIMSS Fourth Grade 


The results of the fourth-grade analyses are summarized in table 3-17. 
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Table 3-17. Characteristics with p values less than .05, U.S. TIMSS fourth-grade schools: 2011 


Analysis 

Characteristics with p values less than .05 

Original sample 
Model a 
Model b 

School control; community type; fourth-grade enrollment; Black, non- 
Hispanic; Hispanic; American Indian or Alaska Native 
Private school; South region; total school enrollment; fourth-grade 
enrollment; Hispanic; American Indian or Alaska Native 
Private school; total school enrollment; fourth-grade enrollment 

Sample with substitutes 

Model a 
Model b 

School control; fourth-grade enrollment; Black, non-Hispanic; Hispanic; 
American Indian or Alaska Native 

Private school; total school enrollment; American Indian or Alaska Native 
Private school; total school enrollment; fourth-grade enrollment 

Nonresponse adjusted 

Census region; Black, non-Hispanic 


For the original sample of schools in TlMSS-4 in the United States, six variables were found to be 
statistically significantly related to participation in the bivariate analysis: school control; community type; 
fourth-grade enrollment; the percentage of Black, non-Hispanic students; the percentage of Hispanic 
students; and the percentage of American Indian or Alaska Native students. Although each of these 
findings indicates some potential for nonresponse bias, when all of these factors were considered 
simultaneously in a regression analysis, private schools. South region, total school enrollment, fourth- 
grade enrollment, the percentage of Hispanic students, and the percentage of American Indian or Alaska 
Native students were significant predictors of participation. The second model showed that private 
schools, total school enrollment, and fourth-grade enrollment were significant predictors of participation 
(with summed race/ethnicity percentage). 


For the final sample of schools in TlMSS-4, five of the six variables remained statistically significant in 
the bivariate analysis: school control; fourth-grade enrollment; the percentage of Black, non-Hispanic 
students; the percentage of Hispanic students; and the percentage of American Indian or Alaska Native 
students. When all of these factors were considered simultaneously in a regression analysis, private 
schools, total school enrollment and the percentage of American Indian or Alaska Native students 
remained significant predictors of participation. The second model showed that private schools, total 
school enrollment, and fourth-grade enrollment were significant predictors of participation (with summed 
race/ethnicity percentage). 


For the final sample of schools in TlMSS-4 with school nonresponse adjustments applied to the weights, 
only two variables were statistically significant in the bivariate analysis: region and the percentage of 
Black, non-Hispanic students. The multivariate regression analysis cannot be conducted after the school 
nonresponse adjustments are applied to the weights. 
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These results suggest that there is some potential for nonresponse bias in the U.S. TlMSS-4 original 
sample based on the characteristics studied. It also suggests that, while there is no evidence that the use of 
substitute schools reduced the potential for bias, it has not added to it substantially. The application of 
school nonresponse adjustments substantially reduced the potential for bias. 


Results for Original Respondent Sample — PIRLS Fourth Grade 

This section presents the results of the nonresponse bias analysis, based exclusively on the original 
sample of 437 eligible U.S. schools for PlRLS-4. The distribution of the responding original school 
sample was compared with that of the total eligible original school sample using base weights in each 
case. All original schools in the sample that declined to participate in the survey were treated as 
nonparticipants regardless of whether they were substituted by a substitute school. The weighted and 
unweighted response rates were 80 percent, with 349 out of 437 eligible schools participating. See 
table 3-2 for details on the PlRLS-4 school participation rates. 

Categorical variables (PIRLS-4), The distribution of participating and eligible schools in the U.S. 
PlRLS-4 sample by the four characteristics is shown in table 3-18. Based on these analyses, the chi- 
square statistics for school control and community type are significant and suggests that there is evidence 
of relationships with participating in the assessment. In particular, public schools were overrepresented 
among participating schools (94.6 vs. 91.3 percent, respectively), and private schools were 
underrepresented among participating schools (5.4 vs. 8.7 percent, respectively). Similarly, schools in 
cities were underrepresented among participating schools relative to eligible schools (27.2 vs. 30.2 
percent, respectively), while schools in rural areas were overrepresented among participating schools 
(23.8 vs. 22.6 percent, respectively). There are no statistically significant relationships between 
participation status and any of the other characteristics shown in table 3-18. 
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Table 3-18. Percentage distribution of eligible and participating schools in the U.S. PIRLS fourth-grade 
original sample, by selected categorical variables: 2011 


School characteristic 

Sample schools 
Eligible Participating 

(percent) (percent) 

(N = 437) (N = 349) 

Bias 

Relative 

bias 

Chi- 
square p 
value 

School control 
Public 

91.3 

94.6 

3.24 

0.035 

0.000 

Private 

8.7 

5.4 

-3.24 

-0.374 


Community type 
City 

30.2 

27.2 

-3.01 

-0.100 

0.027 

Suburban 

35.5 

36.7 

1.15 

0.032 


Town 

11.7 

12.3 

0.64 

0.055 


Rural 

22.6 

23.8 

1.22 

0.054 


Region 

Northeast 

16.4 

15.0 

-1.38 

-0.084 

0.163 

Midwest 

21.2 

20.4 

-0.81 

-0.038 


South 

38.4 

40.7 

2.38 

0.062 


West 

24.0 

23.9 

-0.19 

-0.008 


Poverty level 
High 

39.9 

38.7 

-1.20 

-0.030 

0.328 

Low 

60.1 

61.3 

1.20 

0.020 



NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Eligible schools contained at 
least one fourth-grade class. Participating schools agreed to have their students assessed. The bias is the 
difference between the respective estimates for the participants and the eligible sample. The relative bias 
is calculated as the bias divided by the estimate from the eligible sample. Schools were weighted by their 
school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA) Progress in 
International Reading Eiteracy Study (PIRLS), 2011. 


Continuous variables (PIRLS-4), Summary means for each continuous variable for participating and 
eligible schools are shown in tables 3-19 through 3-21. Twenty-four of the 399 public schools had a 
missing value for the free or reduced-price lunch variable; these schools were excluded from the analysis. 


Participating schools had a higher mean fourth-grade enrollment than the eligible sample (99. 1 vs. 94.0, 
respectively; table 3-19). There were no statistically significant differences detected between the 
participating and eligible public schools for race/ethnicity (table 3-20). Participating schools had a lower 
mean percentage than the eligible sample of students eligible for free or reduced-price lunch (46.6 vs. 
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48.1 percent, respectively; table 3-21). However, this must be interpreted with caution because this 
variable was missing for 24 public schools. 


Table 3-19. Mean enrollment of eligible and participating schools in the U.S. PIRLS fourth-grade 
original sample: 2011 



Sample schools 




Student 

Eligible 

(mean) 

Participating 

(mean) 


Relative 

i test 

enrollment 

(N = 437) 

(N = 349) 

Bias 

bias 

p value 

Total school 

561.3 

572.1 

10.75 

0.019 


Fourth grade 

94.0 

99.1 

5.07 

0.054 



NOTE: Eligible schools contained at least one fourth-grade class. Participating schools agreed to have 
their students assessed. The bias is the difference between the respective estimates for the participants and 
the eligible sample. The relative bias is calculated as the bias divided by the estimate from the eligible 
sample. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Eiteracy Study (PIRLS), 20 1 E 


Table 3-20. Mean percentage of students in eligible and participating schools in the U.S. PIRLS fourth- 
grade original sample, by race/ethnicity: 2011 


Sample schools 


Race/ethnicity 

Eligible 
(percent) 
(N = 437) 

Participating 
(percent) 
(N = 349) 

Bias 

Relative 

bias 

i test 
p value 

White, non-Hispanic 

55.9 

56.3 

0.39 

0.007 

0.646 

Black, non-Hispanic 

15.3 

13.8 

-1.51 

-0.098 

0.052 

Hispanic 

23.5 

24.5 

1.05 

0.045 

0.098 

Asian or Pacific Islander 

4.5 

4.5 

-0.03 

-0.007 

0.829 

American Indian or Alaska Native 

0.8 

0.9 

0.10 

0.119 

0.060 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Eligible schools contained at least one fourth-grade class. Participating schools agreed to 
have their students assessed. The bias is the difference between the respective estimates for the 
participants and the eligible sample. The relative bias is calculated as the bias divided by the estimate 
from the eligible sample. Schools were weighted by their school base weights that did not include a 
nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 
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Table 3-21. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. PIRLS fourth-grade original sample: 2011 



Sample schools 



Students 

Eligible 
(percent) 
(N = 375) 

Participating 
(percent) 
(N = 308) 

Relative 
Bias bias 

t test 
p value 

Percentage of students eligible for 
free or reduced-price lunch 

48.1 

46.6 

-1.50 -0.031! 

0.050! 


! Interpret data with caution. 

NOTE: Information on percentage of students eligible for free or reduced-price lunch is missing for 24 of 
the 399 public schools in the original sample and 22 of the 330 public schools that participated. Eligible 
schools contained at least one fourth-grade class. Participating schools agreed to have their students 
assessed. The bias is the difference between the respective estimates for the participants and the eligible 
sample. The relative bias is calculated as the bias divided by the estimate from the eligible sample. 

Schools were weighted by their school base weights that did not include a nonresponse adjustment factor. 
SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Eiteracy Study (PIRLS), 2011. 


Logistic regression model (PIRLS-4), To examine the joint relationship of various characteristics to 
school nonresponse, the analysis utilized a logistic regression model with participation status as the binary 
dependent variable and frame characteristics as predictor variables. Public and private schools were 
modeled together using the variables available for all schools. 


Standard errors and tests of hypotheses for the full model parameter estimates are presented in tables 3- 
22a (with four race/ethnicity variables) and 3 -22b (with summed race/ethnicity percentage). Private 
schools, high poverty schools, total school enrollment, and fourth-grade enrollment are significant 
predictors of school participation in table 3 -22a. The negative parameter estimates indicate that, relative 
to public and low-poverty schools, private and high poverty schools, respectively, were somewhat 
underrepresented among the participating schools, and the total enrollment in participating schools was 
smaller than in all eligible schools (i.e., the smaller the total enrollment, the more likely a school was to 
participate). The positive parameter estimates indicate that the fourth-grade enrollment in participating 
schools was larger than in all eligible schools. The A test statistic to determine whether the race/ethnicity 
characteristics are simultaneously equal to 0 was 3.69 with a p value of 0.0002, which indicates a 
significant relationship detected with participation. 
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Table 3 -22a. Logistic regression model parameter estimates (with four race/ethnicity variables) using 
the U.S. PIRLS fourth-grade original sample (N = 437): 2011 


Parameter 

Parameter 

estimate 

Standard t test for Hq: 

error parameter = 0 

p value 

Intercept 

1.994 

0.4809 

4.1461 

0.0001 

City 

-0.463 

0.3497 

-1.3243 

0.1894 

Suburban 

-0.113 

0.3526 

-0.3204 

0.7496 

Town 

-0.138 

0.4843 

-0.2849 

0.7765 

Private school 

-1.311 

0.5262 

-2.4916 

0.0149 

High poverty 

-0.889 

0.3857 

-2.3054 

0.0239 

Northeast 

-0.127 

0.4323 

-0.2937 

0.7698 

Midwest 

0.007 

0.4309 

0.0153 

0.9878 

South 

0.662 

0.3870 

1.7108 

0.0913 

Total school enrollment 

-0.002 

0.0008 

-2.0514 

0.0437 

Fourth-grade enrollment 

0.014 

0.0067 

2.1045 

0.0387 

Black, non-Hispanic 

-0.003 

0.0066 

-0.5334 

0.5953 

Hispanic 

0.012 

0.0067 

1.7325 

0.0873 

Asian or Pacific Islander 

-0.004 

0.0149 

-0.2460 

0.8063 

American Indian or Alaska Native 

0.160 

0.1117 

1.4317 

0.1564 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Black includes African 
American, and Hispanic includes Fatino. Racial categories exclude Hispanic origin. Schools were 
weighted by their school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 


Private schools, high poverty schools, and fourth-grade enrollment again are significant predictors of 
school participation in table 3-22b. This model also shows that the summed race/ethnicity percentage is 
not significantly related to participation. 
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Table 3-22b. Logistic regression model parameter estimates (with summed race/ethnicity percentage) 
using the U.S. PIRLS fourth-grade original sample (N = 437): 2011 


Parameter 

Parameter 

estimate 

Standard error 

t test for Hq: 
parameter = 0 

p value 

Intercept 

2.181 

0.4437 

4.9158 

0.0000 

City 

-0.544 

0.3452 

-1.5763 

0.1192 

Suburban 

-0.145 

0.3539 

-0.4106 

0.6825 

Town 

-0.041 

0.4812 

-0.0852 

0.9323 

Private school 

-1.179 

0.5243 

-2.2481 

0.0275 

High poverty 

-0.752 

0.3677 

-2.0450 

0.0444 

Northeast 

-0.470 

0.3968 

-1.1856 

0.2395 

Midwest 

-0.265 

0.4015 

-0.6603 

0.5111 

South 

0.279 

0.3346 

0.8325 

0.4077 

Total school enrollment 

-0.002 

0.0008 

-1.8862 

0.0631 

Fourth-grade enrollment 

0.015 

0.0069 

2.2354 

0.0284 

Summed race/ethnicity percentage 

0.003 

0.0054 

0.6334 

0.5284 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing.) Summed race/ethnicity 
percentage includes Black, non-Hispanic; Hispanic; Asian or Pacific Islander; and American Indian or 
Alaska Native. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Eiteracy Study (PIRLS), 2011. 


Results for Respondent Sample With Substitutes (Final Sample) — PIRLS 
Fourth Grade 

This section presents the nonresponse bias analysis based on the final sample of 437 eligible schools for 
PIRLS-4. The distribution of the responding sample, including participating substitutes, was compared to 
the total eligible final sample. School base weights were used for both the eligible sample and the 
participating schools. Only eligible original schools that refused and were not successfully replaced by a 
substitute were treated as nonparticipants. All other eligible original sample schools were treated as 
participating. Through the use of substitutes, the weighted and unweighted school response rates for 
PIRLS-4 were 85 percent, with 370 out of 437 schools participating. 

Categorical variables (PIRLS-4). The distribution of participating and eligible schools by the four 
characteristics is shown in table 3-23. Only school control was found to be statistically significant among 
the categorical variables. In particular, public schools were overrepresented among participating schools 
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(94.9 vs. 91.3 percent, respectively), and private schools were underrepresented among participating 
schools (5. 1 vs. 8.7 percent, respectively). These differences were slightly larger than that shown in 
table 3-18, in which only the original sample was considered. Thus while there is no evidence that the use 
of substitute schools eliminated the potential for bias, as indicated by this variable, it certainly has also 
not substantially added to it. There are no statistically significant relationships detected between 
participation status and the other characteristics shown in table 3-23. 


Table 3-23. Percentage distribution of eligible and participating schools in the U.S. PIRLS fourth-grade 
final sample, by selected categorical variables: 2011 


School characteristic 

Sample schools 
Eligible Participating 
(percent) (percent) 

(N = 437) (N = 370) 

Bias 

Relative 

bias 

Chi-square 
p value 

School control 





0.000 

Public 

91.3 

94.9 

3.55 

0.039 


Private 

8.7 

5.1 

-3.55 

-0.409 


Community type 





0.239 

City 

30.2 

28.4 

-1.85 

-0.061 


Suburban 

35.5 

36.5 

0.96 

0.027 


Town 

11.7 

11.9 

0.21 

0.018 


Rural 

22.6 

23.3 

0.68 

0.030 


Region 





0.415 

Northeast 

16.4 

15.2 

-1.14 

-0.070 


Midwest 

21.2 

20.9 

-0.35 

-0.016 


South 

38.4 

39.2 

0.88 

0.023 


West 

24.0 

24.7 

0.61 

0.025 


Poverty level 





0.881 

High 

39.9 

39.8 

-0.15 

-0.004 


Low 

60.1 

60.2 

0.15 

0.003 



NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the ifee or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Eligible schools contained at 
least one fourth-grade class. Participating schools agreed to have their students assessed. The bias is the 
difference between the respective estimates for the participants and the eligible sample. The relative bias 
is calculated as the bias divided by the estimate from the eligible sample. Schools were weighted by their 
school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 
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Continuous variables (PIRLS-4), Summary means for each continuous variable for participating and 
eligible schools are shown in tables 3-24 through 3-26. Twenty-four of the 399 public schools had a 
missing value for the free or reduced-price lunch variable; these schools were excluded from the analysis. 


Participating schools had a higher mean fourth-grade enrollment than the eligible sample (99. 1 vs. 94.0, 
respectively; table 3-24). Participating schools had a larger mean percentage than the eligible sample of 
Hispanic students (24.4 vs. 23.1 percent, respectively; table 3-25). The difference in the fourth-grade 
enrollment remained the same as shown in table 3-19, in which only the original sample was considered. 
The differences in the percentage of students who are Hispanic are somewhat larger than that shown in 
table 3-20, in which only the original sample was considered. Thus, as in the case with school control, 
while there is no evidence that the use of substitute schools substantially reduced the potential for bias, as 
indicated by these variables, it has also not substantially added to it. 


There was no statistically significant difference detected between the participating and eligible public 
schools for free or reduced-price lunch (table 3-26). However, this must be interpreted with caution 
because the variable is missing for 24 schools. 


Table 3-24. Mean enrollment of eligible and participating schools in the U.S. PIRLS fourth-grade final 
sample: 2011 



Sample schools 




Student enrollment 

Eligible 
(mean) 
(N = 437) 

Participating 
(mean) 
(N = 370) 

Bias 

Relative 

bias 

i test 
p value 

Total school 

562.2 

575.2 

13.00 



Fourth grade 

94.0 

99.1 

5.07 




NOTE: Eligible schools contained at least one fourth-grade class. Participating schools agreed to have 
their students assessed. The bias is the difference between the respective estimates for the participants and 
the eligible sample. The relative bias is calculated as the bias divided by the estimate from the eligible 
sample. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Eiteracy Study (PIRLS), 2011. 
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Table 3-25. Mean percentage of students in eligible and participating schools in the U.S. PIRLS fourth- 
grade final sample, by race/ethnicity: 2011 


Sample schools 


Race/ethnicity 

Eligible 
(percent) 
(N = 437) 

Participating 
(percent) 
(N = 370) 

Bias 

Relative 

bias 

t test 
p value 

White, non-Hispanic 

56.3 

56.1 

-0.22 

0.00 

0.7839 

Black, non-Hispanic 

15.2 

13.9 

-1.28 

-0.08 

0.0752 

Hispanic 

23.1 

24.4 

1.33 

0.06 

0.0176 

Asian or Pacific Islander 

4.7 

4.8 

0.09 

0.02 

0.4668 

American Indian or Alaska Native 

0.8 

0.9 

0.07 

0.09 

0.0575 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Eligible schools contained at least one fourth-grade class. Participating schools agreed to 
have their students assessed. The bias is the difference between the respective estimates for the 
participants and the eligible sample. The relative bias is calculated as the bias divided by the estimate 
from the eligible sample. Schools were weighted by their school base weights that did not include a 
nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 


Table 3-26. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. PIRLS fourth-grade final sample: 2011 



Sample schools 



Students 

Eligible 
(percent) 
(N = 375) 

Participating 
(percent) 
(N = 328) 

Relative 
Bias bias 

t test 
p value 

Percentage of students eligible for 
free or reduced-price lunch 

47.9 

46.7 

-1.20 -0.025 

0.059 


NOTE: Information on percentage of students eligible for free or reduced-price lunch is missing for 24 of 
the 399 public schools in the final sample and 23 of the 351 public schools that participated. Eligible 
schools contained at least one fourth-grade class. Participating schools agreed to have their students 
assessed. The bias is the difference between the respective estimates for the participants and the eligible 
sample. The relative bias is calculated as the bias divided by the estimate from the eligible sample. 

Schools were weighted by their school base weights that did not include a nonresponse adjustment factor. 
SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 


Logistic regression model (PIRLS-4), To examine the joint relationship of various characteristics to 
school nonresponse, the analysis utilized a logistic regression model with participation status as the binary 
dependent variable and frame characteristics as predictor variables. Six schools were excluded from the 
analysis due to missing information for race/ethnicity and total school enrollment. 
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Private schools and fourth-grade enrollment remained significant predictors of school participation in 
table 3 -27a. The negative parameter estimates indicate that, relative to public schools, private schools 
were somewhat underrepresented among the participating schools. The positive parameter estimates 
indicate that total enrollment in participating schools was larger than in all eligible schools, i.e., the larger 
the total enrollment, the more likely a school was to participate. The F test statistic to determine whether 
the race/ethnicity characteristics are simultaneously equal to 0 was 3.27 with a p value of 0.0006, which 
indicates a significant relationship detected with participation. 


Private schools and fourth-grade enrollment remained significant predictor of school participation in 
table 3-27b. This model also shows that the summed race/ethnicity percentage is not significantly related 
to participation. 


Table 3 -27a. Logistic regression model parameter estimates (with four race/ethnicity variables) in the 
U.S. PIRLS fourth-grade final sample (N = 437): 2011 


Parameter 

Parameter 

estimate 

Standard 

error 

t test for Ho: 
parameter = 0 

p value 

Intercept 

2.082 

0.5995 

2.A111 

0.0009 

City 

-0.364 

0.4363 

-0.8337 

0.4071 

Suburban 

-0.105 

0.4455 

-0.2366 

0.8136 

Town 

-0.347 

0.5378 

-0.6457 

0.5205 

Private school 

-1.366 

0.5915 

-2.3099 

0.0236 

High poverty 

-0.772 

0.4488 

-1.7203 

0.0895 

Northeast 

-0.327 

0.5313 

-0.6147 

0.5406 

Midwest 

-0.027 

0.5720 

-0.0476 

0.9621 

South 

0.237 

0.4942 

0.4796 

0.6329 

Total school enrollment 

-0.002 

0.0011 

-1.8812 

0.0638 

Fourth-grade enrollment 

0.023 

0.0106 

2.1802 

0.0324 

Black, non-Hispanic 

-0.004 

0.0073 

-0.5402 

0.5906 

Hispanic 

0.012 

0.0088 

1.4123 

0.1620 

Asian or Pacific Islander 

-0.001 

0.0164 

-0.0742 

0.9411 

American Indian or Alaska Native 

0.154 

0.1156 

1.3347 

0.1860 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the ifee or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Black includes African 
American, and Hispanic includes Fatino. Racial categories exclude Hispanic origin. Schools were 
weighted by their school base weights that did not include a nonresponse adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 
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Table 3-27b. Logistic regression model parameter estimates (with summed race/ethnicity percentage) in 
the U.S. PIRLS fourth-grade final sample (N = 437): 2011 


Parameter 

Parameter 

estimate 

Standard t test for Hq: 

error parameter = 0 

p value 

Intercept 

2.251 

0.5648 

3.9858 

0.0002 

City 

-0.473 

0.4378 

-1.0802 

0.2835 

Suburban 

-0.128 

0.4531 

-0.2832 

0.7778 

Town 

-0.262 

0.5367 

-0.4881 

0.6269 

Private school 

-1.194 

0.5920 

-2.0166 

0.0473 

High poverty 

-0.650 

0.4450 

-1.4597 

0.1486 

Northeast 

-0.674 

0.4775 

-1.4117 

0.1622 

Midwest 

-0.317 

0.5098 

-0.6220 

0.5358 

South 

-0.164 

0.4054 

-0.4051 

0.6865 

Total school enrollment 

-0.002 

0.0011 

-1.8638 

0.0663 

Fourth-grade enrollment 

0.026 

0.0107 

2.4145 

0.0182 

Summed race/ethnicity percentage 

0.003 

0.0062 

0.5235 

0.6022 


NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Summed race/ethnicity 
percentage includes Black, non-Hispanic; Hispanic; Asian or Pacific Islander; and American Indian or 
Alaska Native. Schools were weighted by their school base weights that did not include a nonresponse 
adjustment factor. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Eiteracy Study (PIRLS), 2011. 


Results for Nonresponse-Adjusted Sample With Substitutes (Nonresponse- 
Adjusted Sample) — PIRLS Fourth Grade 

This section presents the nonresponse bias analysis based on the final sample of 437 eligible schools for 
PlRLS-4. The distribution of the responding sample, including participating substitutes, was compared to 
the total eligible final sample just like the previous section. However, in this section, school base weights 
were used for the eligible sample of schools, whereas nonresponse-adjusted weights were used for the 
participating schools. Only eligible original schools that refused and were not successfully replaced by a 
substitute were treated as nonparticipants. All other eligible original sample schools were treated as 
participating. 

Categorical variables (PIRLS-4). The distribution of participating and eligible schools by the four 
characteristics is shown in table 3-28. No variables were found to be statistically significant among the 
categorical variables. 
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Table 3-28. Percentage distribution of eligible and participating schools in the U.S. PIRLS fourth-grade 
nonresponse-adjusted sample, by selected categorical variables: 2011 


School characteristic 

Sample schools 
Eligible Participating 
(percent) (percent) 

(N = 437) (N = 370) 

Bias 

Relative 

bias 

Chi-square 
p value 

School control 





0.995 

Public 

91.3 

91.3 

0.00 

0.000 


Private 

8.7 

8.7 

0.00 

0.000 


Community type 





0.431 

City 

30.2 

28.9 

-1.33 

-0.044 


Suburban 

35.5 

36.4 

0.91 

0.026 


Town 

11.7 

11.9 

0.24 

0.020 


Rural 

22.6 

22.8 

0.19 

0.008 


Region 





0.110 

Northeast 

16.4 

16.3 

-0.03 

-0.002 


Midwest 

21.2 

22.0 

0.79 

0.037 


South 

38.4 

38.1 

-0.29 

-0.008 


West 

24.0 

23.6 

-0.47 

-0.020 


Poverty level 





0.872 

High 

39.9 

39.9 

-0.06 

-0.002 


Low 

60.1 

60.1 

0.06 

0.001 



NOTE: For public schools, a high poverty school is defined as one in which 50 percent or more of the 
students are eligible for participation in the free or reduced-price lunch (FRPL) program, and low poverty 
is defined as having less than 50 percent eligible. All private schools are treated as low-poverty schools. 
For definitions of these urban-centric locales, see http://nces.ed.gov/survevs/urbaned/deFinitions.asp . 
Region is the Census region of the country (see section 3.5 for state listing). Eligible schools contained at 
least one fourth-grade class. Participating schools agreed to have their students assessed. The bias is the 
difference between the respective estimates for the participants and the eligible sample. The relative bias 
is calculated as the bias divided by the estimate from the eligible sample. Schools were weighted by their 
school nonresponse adjusted weight. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 


Continuous variables (PIRLS-4), Summary means for each continuous variable for participating and 
eligible schools are shown in tables 3-29 through 3-31. Twenty-four of the 399 public schools had a 
missing value for the free or reduced-price lunch variable; these schools were excluded from the analysis. 


There were no statistically significant differences detected between participating and eligible schools for 
enrollment (table 3-29) or race/ethnicity (table 3-30). There was also no statistically significant difference 
detected between the participating and eligible public schools for free or reduced-price lunch (table 3-31). 
However, this must be interpreted with caution because the variable is missing for 24 schools. 
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Table 3-29. Mean enrollment of eligible and participating schools in the U.S. PIRLS fourth-grade 
nonresponse-adjusted sample: 2011 



Sample schools 




Student enrollment 

Eligible 
(mean) 
(N = 437) 

Participating 
(mean) 
(N = 370) 

Bias 

Relative 

bias 

t test 
p value 

Total school 

562.2 

565.2 

2.98 


0.776 

Fourth grade 

94.0 

96.2 

2.19 


0.096 


NOTE: Eligible schools contained at least one fourth-grade class. Participating schools agreed to have 
their students assessed. The bias is the difference between the respective estimates for the participants and 
the eligible sample. The relative bias is calculated as the bias divided by the estimate from the eligible 
sample. Schools were weighted by their school nonresponse adjusted weight. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA Progress in 
International Reading Eiteracy Study (PIRLS), 2011. 


Table 3-30. Mean percentage of students in eligible and participating schools in the U.S. PIRLS fourth- 
grade nonresponse-adjusted sample, by race/ethnicity: 2011 


Sample schools 


Race/ethnicity 

Eligible 
(percent) 
(N = 437) 

Participating 
(percent) 
(N = 370) 

Bias 

Relative 

bias 

t test 
p value 

White, non-Hispanic 

56.3 

56.2 

-0.04 

0.00 

0.968 

Black, non-Hispanic 

15.2 

14.0 

-1.20 

-0.08 

0.212 

Hispanic 

23.1 

24.2 

1.13 

0.05 

0.232 

Asian or Pacific Islander 

4.7 

4.7 

0.00 

0.00 

0.991 

American Indian or Alaska Native 

0.8 

0.9 

0.11 

0.13 

0.566 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Eligible schools contained at least one fourth-grade class. Participating schools agreed to 
have their students assessed. The bias is the difference between the respective estimates for the 
participants and the eligible sample. The relative bias is calculated as the bias divided by the estimate 
from the eligible sample. Schools were weighted by their school nonresponse adjusted weight. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 
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Table 3-31. 


Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. PIRLS fourth-grade nonresponse-adjusted sample: 


2011 



Sample schools 



Students 

Eligible 
(percent) 
(N = 375) 

Participating 
(percent) 
(N = 328) 

Relative 
Bias bias 

t test 
p value 

Percentage of students eligible for 
free or reduced-price lunch 

47.9 

47.4 

-0.50 -0.010 

0.468 


NOTE: Information on percentage of students eligible for free or reduced-price lunch is missing for 24 of 
the 399 public schools in the final sample and 23 of the 351 public schools that participated. Eligible 
schools contained at least one fourth-grade class. Participating schools agreed to have their students 
assessed. The bias is the difference between the respective estimates for the participants and the eligible 
sample. The relative bias is calculated as the bias divided by the estimate from the eligible sample. 

Schools were weighted by their school nonresponse adjusted weight. 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Progress in 
International Reading Eiteracy Study (PIRLS), 2011. 


Summary — PIRLS Fourth Grade 


The results of the grade 4 analyses are summarized in table 3-32. 


Table 3-32. Characteristics with p values less than .05, U.S. PIRLS fourth-grade schools: 2011 


Analysis 

Characteristics with p values less than .05 

Original sample 
Model a 
Model b 

School control, community type, fourth-grade enrollment, students eligible 
for free or reduced-price lunch 

Private school, high poverty, total school enrollment, fourth-grade 
enrollment 

Private school, high poverty, fourth-grade enrollment 

Final sample 
Model a 
Model b 

School control, fourth-grade enrollment, Hispanic 
Private school, fourth-grade enrollment 
Private school, fourth-grade enrolment 


For the original sample of schools in PlRLS-4 in the United States, four variables were found to be 
statistically significantly related to participation in the bivariate analysis: school control (table 3-18); 
community type (table 3-18); fourth grade enrollment (table 3-19); and the percentage of students eligible 
for free or reduced-price lunch (table 3-21). Although each of these findings indicates some potential for 
nonresponse bias, when all of these factors were considered simultaneously in a regression analysis, 
private schools, high-poverty schools, total school enrollment, and fourth-grade enrollment were 
significant predictors of participation (table 3-22a). The second model showed that private schools, total 
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high-poverty schools, and fourth grade enrollment were significant predictors of participation 
(table 3-22b, with summed race/ethnicity percentage). 

For the final sample of schools in PlRLS-4, two of the four variables remained statistically significant in 
the bivariate analysis: school control (table 3-23) and fourth-grade enrollment (table 3-24). Additionally, 
the percentage of Hispanic students was statistically significantly related to participation (table 3-25). 
When all of these factors were considered simultaneously in a regression analysis, private schools and 
fourth-grade enrollment remained significant predictors of participation in both models (tables 3 -27a and 
3 -27b, with summed race/ethnicity percentage). 

For the final sample of schools in PlRLS-4 with school nonresponse adjustments applied to the weights, 
no variables were statistically significant in the bivariate analysis: The multivariate regression analysis 
cannot be conducted after the school nonresponse adjustments are applied to the weights. 

These results suggest that there is some potential for nonresponse bias in the U.S. PlRLS-4 original 
sample based on the characteristics studied. It also suggests that, while there is no evidence that the use of 
substitute schools reduced the potential for bias, it has not added to it substantially. The application of 
school nonresponse adjustments completely reduced the potential for bias, based on the characteristics 
studied. 


Conclusions 

The investigation into nonresponse bias at the school level for U.S. 2011 TIMSS sample for grade 4 and 
the PIRLS sample for grade 4 has shown that there was no statistically significant relationship detected 
between participation status and the majority of school characteristics that are available for analysis. It 
also suggests that, while there is no evidence that the use of substitute schools reduced the potential for 
bias, it has not added to it substantially. The application of school nonresponse adjustments substantially 
reduced the potential for bias. However, the results do indicate some potential for bias in the data arising 
from school control, enrollment, and regional differences in participation, along with the fact that schools 
with higher percentages of minority students were less likely to participate. The results vary by sample 
and grade as summarized earlier. Since TIMSS and PIRLS are conducted under a set of standard rules 
designed to facilitate international comparisons, the U.S. nonresponse bias analysis results were not used 
to adjust the U.S. data for this source of bias. 
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3.5 


Technical Notes 


Description of Variables 

Frame characteristics for public schools were taken from the 2007-08 Common Core of Data (CCD) and, 
for private schools, from the 2007-08 Private School Survey (PSS). 

Race/ethnicity, Students’ race/ethnicity was obtained through student responses to a two-part question. 
Students were asked first whether they were Hispanic or Latino and then asked whether they were 
members of the following racial groups: American Indian/Alaska Native; Asian; Black or African 
American; Native Hawaiian or other Pacific Islander; or White. Multiple responses to the race 
classification question were allowed. 

Community type. Community type was derived from the locale variable based on how the school is 
situated in a particular location relative to populous areas, based on the school’s address. 

City consists of territory inside an urbanized area and inside a principal city with population of 
250,000 or more, territory inside an urbanized area and inside a principal city with population less 
than 250,000 and greater than or equal to 100,000, and territory inside an urbanized area and inside 
a principal city with population less than 100,000. 

Suburb consists of territory outside a principal city and inside an urbanized area with population of 
250,000 or more, territory outside a principal city and inside an urbanized area with population less 
than 250,000 and greater than or equal to 100,000, and territory outside a principal city and inside 
an urbanized area with population less than 100,000. 

Town consists of territory inside an urban cluster that is less than or equal to 10 miles from an 
urbanized area, territory inside an urban cluster that is more than 1 0 miles and less than or equal to 
35 miles from an urbanized area, and territory inside an urban cluster that is more than 35 miles of 
an urbanized area. 

Rural consists of Census-defined rural territory that is less than or equal to 5 miles from an 
urbanized area, as well as rural territory that is less than or equal to 2.5 miles from an urban cluster. 
Census-defined rural territory that is more than 5 miles but less than or equal to 25 miles from an 
urbanized area, as well as rural territory that is more than 2.5 miles but less than or equal to 10 
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miles from an urban cluster, and Census-defined rural territory that is more than 25 miles from an 
urbanized area and is also more than 1 0 miles from an urban cluster. 

Region. Region is the Census region of the country. Northeast consists of Connecticut, Maine, 
Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, and Vermont. 
Midwest consists of Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North 
Dakota, Ohio, South Dakota, and Wisconsin. South consists of Alabama, Arkansas, Delaware, District of 
Columbia, Florida, Georgia, Kentucky, Louisiana, Maryland, Mississippi, North Carolina, Oklahoma, 
South Carolina, Tennessee, Texas, Virginia, and West Virginia. West consists of Alaska, Arizona, 
California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oregon, Utah, Washington, and 
Wyoming. 

Percentage of students eligible for free or reduced-price lunch. The proportion of students in a school 
eligible for the free or reduced-price lunch program, a federally assisted meal program under the National 
School Lunch Act that provides nutritionally balanced, low-cost or free lunches to eligible children each 
school day. The question on the CCD questionnaire asked what percentage of students at the school was 
eligible to receive free or reduced-price lunch through the FRPL around October 1, 2007. It is available 
only for public schools as the PSS data do not provide the same information for private schools. 

Poverty level in public schools. The measure of school poverty is based on the percentage of students 
eligible for FRPL. Schools were classified as “low poverty” if less than 50 percent of the students were 
eligible for FRPL and as “high poverty” if 50 percent or more of the students were eligible. In the interest 
of retaining all of the schools and students in these analyses, private schools were assumed to be low- 
poverty schools-that is, they were assumed to be schools in which less than 50 percent of students were 
eligible for FRPL. 


Statistical Procedures 

Weighting. Before the data are analyzed, responses from the groups of students assessed are assigned 
sampling weights to ensure that their representation in TIMSS and PIRLS 2011 results matches their 
actual percentage of the school population in the grade assessed. 

Responses from the groups of students were assigned sampling weights to adjust for over- or under- 
representation during the sampling of a particular group. The use of sampling weights is necessary for the 
computation of sound, nationally representative estimates. The weight assigned to a student’s responses is 
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the inverse of the probability that the student would be selected for the sample. When responses are 
weighted, none are discarded, and each contributes to the results for the total number of students 
represented by the individual student assessed. Weighting also adjusts for various situations (such as 
school and student nonresponse) because data cannot be assumed to be randomly missing. The 
internationally defined weighting specifications require that each assessed student’s sampling weight 
should be the product of (1) the inverse of the school’s probability of selection, (2) an adjustment for 
school-level nonresponse, (3) the inverse of the classroom’s probability of selection, and (4) an 
adjustment for student-level nonresponse. 

In the analyses in this report, sometimes the appropriate weight (base weight) includes only the 
components of the reciprocals of the respective selection probabilities. This is the case when estimates are 
made based on the entire sample. In other cases nonresponse adjustments, as computed by the 
International Study Center, are also applied. In each case the text and tables make clear which of these 
weighting procedures has been applied. Whereas for substantive analyses using the TIMSS and PIRLS 
data, one would normally apply the nonresponse adjustments when analyzing the data from the 
respondents in the sample, this is not always when the case when carrying out analyses of potential 
nonresponse bias analyses. 

Sampling errors. Sampling errors occur when the discrepancy between a population characteristic and 
the sample estimate arises because not all members of the reference population are sampled for the 
survey. The size of the sample relative to the population and the variability of the population 
characteristics both influence the magnitude of sampling error. The particular sample of students in fourth 
and eighth grade from the 2010-1 1 school year was just one of many possible samples that could have 
been selected. Therefore, estimates produced from the TIMSS and PIRLS sample may differ from 
estimates that would have been produced had another student sample been drawn. This type of variability 
is called sampling error because it arises from using a sample of students in fourth or eighth grade, rather 
than all students in the grade in that year. 

The standard error is a measure of the variability due to sampling when estimating a statistic, and is often 
included in reports containing estimates from survey data. The approach used for calculating sampling 
variances was the jackknife repeated replication (JRR). This report does not show estimates of standard 
errors for each estimate. Rather the effects of sampling error are reflected in the test statistics (for t tests 
and chi-square tests, and t test used in logistic regression analyses) that are presented for each analysis. 
These are described below. 
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The first step to compute the variance with replication is to calculate the estimate of interest from the full 
sample as well as each subsample or replicate. The variation between the replicate estimates and the full- 
sample estimate is then used to estimate the variance for the full sample. Suppose that 9 is the full-sample 
estimate of some population parameter 9. The variance estimator, v(0), takes the form 

g=i 

where 

is the estimate of 9 based on the observations included in the g-th replicate, and 
G is the total number of replicates formed (G = 75 for U.S. TIMSS and PIRLS). 

The standard error is then 

^Jv[9) 

The JRR algorithm used in 201 1 assumes that there are G replicates, each containing two sampled schools 
selected independently. The element 0(g) denotes the estimate using the g-th jackknife replicate. This is 
computed using all cases except those in the g-th replicate of the sample. For those in the g-th replicate, 
the replicate weights for all cases associated with one of the randomly selected units of the pair are 
multiplied by zero, and the replicate weights for the elements associated with the other unit in the 
replicate are doubled. The computation of the JRR variance for any estimate requires the computation of 
the statistic 76 times for any given country: once to obtain the estimate for the full sample, and 75 times 
to obtain the estimate for each of the jackknife replicates (9(g))- 

Tests of Significance 

Comparisons made in the text of this report have been tested for statistical significance. For example, 
when comparing results obtained from the full sample for a given grade, with those obtained only from 
the responding sample units, tests of statistical significance were used to establish whether or not the 
observed differences are statistically significant. The estimation of the standard errors that are required in 
order to undertake the tests of significance is complicated by the complex sample and assessment designs 
which both generate error variance. Together they mandate a set of statistically complex procedures in 
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order to estimate the correct standard errors. As a consequence, the estimated standard errors contain a 
sampling variance component estimated by jackknife repeated replication (JRR). Details on the 
procedures used can be found in the WesVar 4.3 User’s Guide (Westat, 2007). 

Two kinds of statistical tests are included in the report: t tests and chi-square tests. In addition, logistic 
regression analyses were conducted. 


Use of t Tests 

The t test was used for testing for the hypothesis that no difference exists between the means of 
continuous variables for two groups (namely, the full sample and the responding sample). Suppose that 
and Xg are the means for two groups that are being compared and se(xj ~Xg) is the standard error 

of the difference between the means, which accounts for the complex survey design. Then the t test is 
defined as 


se(x^ -Xg)_ 

This statistic is then compared to the critical values of the appropriate student t-distribution to determine 
whether the difference is statistically significant. The appropriate number of degrees of freedom for the 
distribution is given by the number of primary sampling units in the design (in this case the number of 
schools), minus the number of sampling strata. 

Note that this procedure took account of the fact that the two samples in question were not independent 
samples, but in fact the responding sample was a subsample of the full sample. This effect was accounted 
for in calculating the standard error of the difference. Note also that, in those cases where both samples 
were weighted just using base weights, the test is exactly equivalent to testing that the mean of the 
respondents was equal to the mean of the nonrespondents. 

Consider for example the data in table 3-4. The first row shows that the weighted mean total school 
enrollment for the full eligible sample of grade 4 schools is 561.3. For the subsample of schools that 
participated, the corresponding mean is 570.1, resulting in a difference of 8.74. The standard error of this 
estimated difference, calculated so as to reflect the dependency between these two samples, and the 
complex sample design, is 6.65 (not shown in table). This gives rise to a t-statistic of 1 .3 1 (not shown in 
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table), and using 75 degrees of freedom (the appropriate figure for the TIMSS and PIRLS design), the 
resulting significance (or p value) is 0.193. This last figure appears in the table. 

The t test was also used in the logistic regression for testing for the hypothesis for whether each estimated 
parameter estimate is significantly different from 0. Then the t test is defined as 

where bj^ is a parameter estimate and v{b^^ ) is the replication variance estimate for that parameter. This 

statistic is then compared to the critical values of the appropriate student t-distribution, as described 
above, to determine whether the difference is statistically significant. The appropriate number of degrees 
of freedom for the distribution is again given by the number of primary sampling units in the design (in 
this case the number of schools), minus the number of sampling strata. 


Chi-square Tests 

Chi-square tests are used for testing whether two distributions of a given categorical variable are different, 
conducted in a way that reflects the impact of the complex sample design on sampling variance. In this 
instance one distribution is for the full sample and one for the responding sample. Suppose that the 
categorical variable in question has c levels, cross-tabulated producing weighted proportions p. The usual 
Pearson chi-square statistic is calculated as 

PiP if IPiPj 

;=i j=\ / 

where j denotes the categories of the categorical variable, i indexes the samples (full sample and 
respondents), and n indicates the overall sample size. This statistic is not suitable for use directly in a 
statistical test with these data, for two reasons. First, the fact that the respondents are a subset of the full 
sample violates the standard assumptions for a chi-square test of this kind. Second, this statistic does not 
account for the complex sample design used to collect the data. 

Thus the Pearson chi-square statistic is modified appropriately to account for the impact of these two 
features. The resulting test statistic is referred to as the Rao-Scott Adjusted chi-square statistic. It is 
sometimes also referred to as the Satterthwaite -adjusted chi-square statistic. The number of degrees of 
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freedom for the chi-square test, normally given as (c -1), where c is the number of categories of the 
categorical variable for each distribution, is also modified on account of the complex design. The 
modified test statistic is then compared to the chi-square distribution with the appropriate number of 
degrees of freedom, to determine whether the difference in the two distributions is statistically significant. 
For a detailed description of the technique, see Rao and Scott (1984) or Rao and Thomas (2003). 

The first step in the calculation of the Satterthwaite-adjusted chi-square statistic is to form the following 
vector: 



r pu-piPa^ 



II 

Pm -Px P-2 

= 

T2 


V Prc -PrP-cJ 


vT rc J 


An rc X 1 vector made up of the products of the marginal proportions is defined as 


'Pi-Pa'^ 


(pA 

PlP-2 

= 

P2 

yPr-P-cJ 


{Prc, 


For each replicate, an rc x rc matrix is calculated whose f/-th element is made up of 

Ug -yJ{yjg -yj)^ 

where y.^ and yj^ are the i-th and y-th elements of Y calculated for the g-th replicate and and yj are 

the corresponding full-sample values. The i/-th element of the estimated covariance matrix for Y, B = 
cov(Y), is calculated using the following formula: 

^ij = T(yig-yi)(yjg-yj) 

g=i 
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where c is the constant appropriate to the replication method (c = 0.05 for U.S. TIMSS and PIRLS). The 
Satterthwaite’s approximation to degrees of freedom for the chi-square statistic to be calculated is 


^ rc 


V = ■ 


Pi 


rc rc Bjj 

s s — — 

i=\j=\pipj 


Since v will generally not be an integer, interpolation in standard chi-square tables is required. 


Finally, the adjusted chi-square statistic is defined as 


RS3 = 


i=i Pi 


Logistic Regression Modeis 


Let Pi denote the probability that the i-th sampled school will participate. Under the logistic regression 
model, the log odds of response propensity (expressed in terms of the logarithm of Pil{^-pj)), is assumed 
to have the following linear form: 


log 

V 


Pi 

^-Piy 


Pq + Pl^li + Pl^li + - + Pp^ pi 


where X\i, X 2 / ..., Xpi are p auxiliary variables associated with the i-th sampled school, and Pq, 

Pi, ..., pp are coefficients to be estimated. Asymptotic assumptions are used to develop statistical tests to 

determine which, if any, of the coefficients are significantly different from zero. In the analyses in this 
report the standard procedures for carrying out logistic regression analyses have been modified both to 
incorporate the sampling weights in the estimation of the coefficients and to reflect the effect of the 
complex sample design on the variance-covariance matrix of the coefficients. 


The Newton-Raphson algorithm is used to iteratively solve for parameter solutions in the logistic 
regression. Let ^((5) = dL^ (P)/^P be the vector of first partial derivatives of the sample log-likelihood 
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with respect to P . Let H(p) be the matrix of second partial derivatives (or Hessian) of the sample log- 
likelihood having entries , where and are two separate components of P . Denote by 

and the values of q'(p) and H(p) evaluated at , the value of the estimate b at step t. 

The general approach is to approximate the sample log-likelihood at the desired estimate, (b) , at step t 
in the iterative process near the point by a second-order Taylor series expansion: 

4(b) = L„(b^) + q^'(b-b^) + l(b-b^)'tf(b-b^) 


Solving dli /Sb = -I- ^b-b^ j = 0 for b yields the iteration equations 


b*+i=b*- 




assuming has an inverse. Given an initial value for t = 0, the set of iteration equations is solved for 
b^ , b^ is used to solve for b^ , and so on, until the convergence criterion is satisfied. The se[j0) is 

calculated using JRR and repeating the procedure for each replicate. 
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4 . 


Survey Operations 


This chapter describes data collection and related activities for TIMSS and PIRLS 2011 in the United 
States. These activities included recruitment of schools for the national and state samples; sampling of 
students within schools; development of the instruments used; field operations undertaken to administer 
the assessment; post assessment activities associated with scoring and data entry; and several activities 
associated with the preparation of the data to meet international standards. 


4.1 Recruiting Districts and Schoois 

The established protocol for seeking the participation of schools in studies such as TIMSS and PIRLS, 
where participation is voluntary, is to (1) notify state education authorities of the intention to approach 
schools within their jurisdiction, (2) inform authorities at the district level that schools within their 
districts are being sampled, and (3) contact the sampled schools. Participation may be refused at any of 
these levels, so several considerations were important in this context, specifically the need to establish the 
value of participation; establish the timing of the assessment window in conjunction with mandatory 
federal, state, and local assessments; and address concerns about the burden on schools. Private schools 
were contacted directly. In the case of Catholic schools, the diocese was informed and schools were then 
contacted. Non-religious affiliated private schools were contacted directly. 

The recruitment for TIMSS and PIRLS, particularly at grade 8, benefitted from increased involvement at 
the state level. The inclusion of a TIMSS-NAEP linking study at grade 8 allowed NAEP State 
Coordinators within each state to secure participation of sampled public schools in their states. 


Timing of Recruitment Activities 

Assessment dates needed to be established early in the school year in which the assessment was to take 
place or, better still, toward the end of the previous school year. In total, the recruitment phase for TIMSS 
and PIRES 2011 extended from April 2010 through May 201 1, as indicated in exhibit 1-1 in chapter 1. 
States were contacted toward the end of April 2010. Following this, contact with districts began in May 
2010 and continued through the summer. Schools were contacted beginning in June 2010 and activities 
continued through March 2011. 
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The Impact of the Sampling Design on Recruitment Activities 


The sampling design played an important role in the design of recruitment activities. TIMSS and PIRLS 
were administered jointly in the same schools at grade 4, therefore the school sample was slightly larger 
in order to obtain enough students. As a result, 450 schools were sampled in the first instance, along with 
450 first substitutes and 450 second substitutes, for a total of 1,350 schools. The school sample design at 
grade 8 was 600 schools, each with two replacements. The total sample was 1,800 schools, including 
substitutes. In gaining the cooperation of schools, at each grade level the sampled schools were 
approached in the first instance. Operationally this meant first informing the districts in which these 
schools are located and then approaching the schools themselves. If a sampled school refused to 
participate, the district of the first substitute school was approached and the district-school permission 
procedure began anew. If the first substitute school refused as well, then the district of the second 
substitute school was approached and then the second substitute school. The sampling design came into 
play with the distribution of schools across districts and the consequences that this had for the volume and 
length of the permission process. 

With state representatives actively gaining cooperation from the eighth-grade schools (and in some cases 
the fourth-grade schools) in the original national sample, as well as the benchmarking states, the process 
was simplified. In fact, only 22 substitute schools were used at grade 4 and only two substitutes were used 
at grade 8. Thus in both grades, there was an increased original school response rate from previous rounds 
of TIMSS and PIRLS. 

Contacting States 

The chief state school officer and state assessment director in each of the 50 states and the District of 
Columbia were contacted beginning in April 2010. Each person received a combined TIMSS-PIRLS 
package that included an NCES cover letter, instructions on how to obtain the sampled schools in the 
state, a brochure describing the study, a timeline of activities, a summary of activities for the school 
coordinator, and a sheet of frequently asked questions. A copy of the letter sent to states is provided as 
exhibit C-1 in appendix C. Several items of TIMSS and PIRLS information materials (exhibits D-1 
through D-8 in appendix C) were included with the letter. 

Similar packages were supplied to NAEP State Coordinators (NSCs), and NCES and Westat held a series 
of webinars to explain TIMSS at grade 8 and the processes used to report participation and to answer 
questions about the study. In many cases, NSCs created personalized contact letters for districts and 
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schools. Westat tracked cooperation status weekly through a secure automated reporting system used by 
NSCs similar to that used by the Westat recruitment staff working on TIMSS and PIRLS at grade 4. 


Contacting Districts 

After informing the states, similar packages of advance materials were sent to the superintendent and the 
assessment director of each district (or diocese) containing sampled schools. A copy of the letter sent to 
districts is provided as exhibit C-2 in appendix C. Several items of TIMSS and PIRLS information 
materials (exhibits D-1 through D-8 in appendix D) and a list of sampled schools were included with the 
letter. 

During this time, if a sampled school in a cooperating district refused to participate and was judged to be 
a firm refusal, a similar district package was sent out to the district of the first substitute school linked to 
the sampled school. A parallel procedure was adopted with the second substitute district and school in 
those cases where a first substitute school refused to participate. 


Follow-up Contact 

In each case field staff made follow-up calls to district contacts after a few days to discuss the study and 
answer questions. Additionally, 57 school districts required a formal application process and school board 
approval. However, given that many of these were in states with eighth-grade sampled schools, thus states 
were contacting the districts and schools, these requirements were waived. Only 1 6 applications had to be 
formally processed. 

Special security requirements. As a matter of course, each TIMSS field staff member had a current FBI 
clearance and fingerprints on file. In most instances this satisfied school security requirements. However, 
a number of districts required that additional procedures be met before allowing staff to enter schools. For 
example. New York City required staff to obtain additional, local fingerprinting at a local office in New 
York City. Additionally, for each school in New York City, Westat was also required to complete a 
special form with signatures from both the principal and the district superintendent. In Florida, several 
districts required additional fingerprinting and a local security check (Miami-Dade County required the 
completion of an Affidavit of Good Moral Character). In many cases, field staff working in these districts 
had these clearances on file from their recent NAEP work. 
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Contacting Schools 


After district approval was secured, schools were contacted with an initial school information packet. 
Private schools and some parochial schools not linked with a diocese were contacted directly. Each school 
information package was sent on a flow basis governed by receipt of district approval. A copy of the 
cover letter included in the school package is shown as exhibit C-3 in appendix C. Copies of items of 
TIMSS and PIRLS information materials (exhibits D-1 through D-8 in appendix D) were included with 
the letter. 


Follow-up Contact 

After a few days, each school was contacted by a field staff member to discuss the school’s participation 
in TIMSS. In-person visits were made in a small number of schools where efforts to secure participation 
proved difficult. 

Nomination of school coordinator. School principals were asked to identify an individual within the 
school who would act as the TIMSS school coordinator. Principals, deputy principals, teachers, and 
guidance counselors took on this role. School coordinator responsibilities included the following: 

■ Working with Westat to finalize an assessment date; 

■ Preparing a list of all mathematics classrooms in the specified grade; 

■ Identifying excluded classrooms; 

■ Identifying students and teachers in the sampled classrooms; 

■ Identifying excluded students in the sampled classrooms; 

■ Liaising with staff, students, and parents as necessary; 

■ Arranging for space and for the release of students from classes on assessment day; 

■ Ensuring completion and return of the school and teacher questionnaires; and 

■ Holding secure, until 2012 the confidential files that linked student names with IDs 
and then destroying them. 
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Informational Materials and Gifts 


Since the initial contact with states, districts, and schools was by mail, particular attention was paid to 
developing materials that would promote the value of participation and assure all concerned that the 
burden on schools would be minimal. These materials included the following: 


■ TIMSS and PIRLS brochures, a tri-fold describing TIMSS or PIRLS, its history, and 
its importance for the United States (exhibit D-4 and D-8 in appendix D); 

■ TIMSS and PIRLS grade 4 and TIMSS grade 8 Frequently Asked Questions sheets, 
showing a selection of publications arising out of TIMSS and PIRLS (exhibit D.3 and 
D.7 in appendix D); 

■ Timelines of activities listing broad timeframes and activities occurring within them 
for TIMSS and PIRLS grade 4 and TIMSS grade 8 (exhibit D-1 and D-5 in appendix 
D); and 

■ A summary of activities for school coordinators brochure designed to indicate the 
specifics of school participation in TIMSS and PIRLS (exhibit D-2 and D-6 in 
appendix D). 


After the assessment, schools and school coordinators were paid incentives of $200 and $ 1 00, 
respectively. Students each received a clock-compass carabiner. 


Some additional documents were used to maintain contact with schools throughout the year and to inform 
school coordinators of coming TIMSS and PIRLS activities; see, for example, the sheet describing school 
coordinator responsibilities (exhibit D-2 in appendix D). Schools were sent a holiday card toward the end 
of 20 1 0, and at various points school coordinators were sent informational materials to prepare them for 
upcoming TIMSS and PIRLS tasks. Drafts of parent approval letters, forms, and fact sheets were supplied 
to those schools indicating that parent approval was required. Details on the parent approval materials is 
provided in exhibits C-6 through C-14 in appendix C and discussed further below. 


Gaining Cooperation Recruiters (GCRs) 


Nine gaining cooperation recruiters (GCRs) experienced with recruiting districts and schools on previous 
rounds of similar studies (NAEP, PISA, TIMSS, and PIRLS), along with a field manager, were assigned 
to recruit schools at grade 4 as well as private schools at grade 8. On average, each GCR had 
responsibility for about 50 schools. 
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Field manager responsibilities. The field manager engaged in the recruiting phase of the study had the 
following responsibilities: 


■ coordinating recruitment activities of the OCRs within their assignment; 

■ holding weekly one-on-one telephone meetings with their OCRs to monitor progress 
on gaining-cooperation activities and to troubleshoot recruitment strategies; 

■ monitoring and maintaining the Field Management System, ensuring that the 
disposition codes provided by OCRs gave an accurate portrayal of participation status; 
and 

■ acting as troubleshooters to handle special issues that arose during the recruiting 
process. 

GCR responsibilities. The primary responsibility of the nine OCRs was person-to-person interaction 
with the districts and schools along with some more administrative tasks, as follows: 


■ making telephone contact to sampled schools, and in some cases, their districts or 
diocese, within their assignment to obtain permission to conduct TIMSS and PIRLS; 

■ making in-person refusal conversion attempts when necessary and requested by a field 
manager; 

■ completing an electronic record of calls (EROC) of each contact within the Field 
Management System (FMS); 

■ maintaining the most current disposition codes within the FMS; and 

■ meeting weekly with the field manager by conference call to discuss progress. 


Training Field Staff 

A recmitment training workshop for field staff was held at Westat on June 6, 2010. A Gaining 
Cooperation Manual was assembled and sent to OCRs 5 days prior to training. OCRs were asked to 
complete 4 hours of home study before training. 

The training was delivered as distance training through a webinar with the field manager and project 
director. All of the materials that would be sent to district and school contacts in the recruitment package 
were provided and explained. A subsequent discussion ensured that all field staff engaged in the 
recruitment of districts and schools understood the mission of TIMSS and PIRLS, the materials used in 
the United States, and the importance of satisfactory participation rates. 
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The presentation illustrated the responsibilities of a TIMSS and PIRLS recruitment supervisor. 
Preliminary activities, such as planning, organizing, coordinating, and scheduling, were presented as well 
as an examination of the expected administrative tasks. At the conclusion of day 1, field staff were asked 
to familiarize themselves with the TIMSS and PIRLS Frequently Asked Questions and to develop a 
“TIMSS and PIRLS story” of talking points suitable for recruiting tasks. 

Gaining cooperation exercises were performed by the OCRs. Training included open discussions about 
interpersonal skills important for gaining the cooperation of districts and schools. These included 
introducing the assessment; answering questions; dealing with schools’ concerns; overcoming objections; 
and avoiding refusals. Participants were invited to share successful strategies and techniques in past 
recruitment experiences. Also, approaches that might be used with schools for refusal conversion were 
introduced and practiced. To reinforce these presentations and discussions, several role-playing exercises 
were performed after the training session. 

The final activity was the completion of a hands-on tutorial of the TIMSS and PIRLS Field Management 
System (FMS). Each GCR had a laptop workstation connected to the FMS and a set of small exercises to 
complete on dummy case data in the system. 


Monitoring the Recruiting Progress 

Progress in recruiting districts and schools was monitored on a daily basis through the TIMSS and PIRLS 
Field Management System (FMS). GCRs were required to update the FMS for each contact made with 
districts or schools using an electronic record of calls that included updating the disposition code for the 
district/school in question. The disposition code indicated whether the district or school was pending, 
refusing, or cooperating. Using these status codes, the Westat home office tracked the progress of 
recruitment and generated daily reports. These daily reports enabled the operations of recruitment (and 
eventually assessments) to be closely monitored. 

NAEP State Coordinators used a special template similar to the FMS to monitor the recruitment status of 
eighth-grade public schools in their states. 
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Difficulties in Gaining Cooperation 


The principal reasons given by both districts and schools for refusing to participate included, in 
approximate order of priority, the following: 


■ Conflict with mandatory federal, state, and/or local assessments whose outcomes had 
direct implications for districts, schools, teachers and students; 

■ The related matter of the burden that additional testing placed on students at the cost 
of instructional time; and 

■ The limited return on the school’s investment of time since they would not receive 
much usable information on the school, and none on particular students. 


4.2 Sampling Students Within Schools 

Students in participating schools were sampled in a two-stage process. In the first stage, schools were 
asked to provide lists of fourth-grade classrooms or eighth-grade mathematics classrooms that indicated 
the number of students in each class. An equal probability sample of two classrooms (or 
pseudoclassrooms) was identified from the classrooms listed for each school. In the second stage all 
students in sampled classrooms (or pseudoclassrooms) were selected for assessment. 

These procedures are standardized internationally and embodied in software made available to each 
country. The software was developed by the lEA Data Processing Center (DPC) and is known as 
WinWSS (lEA DPC, 2010). The WinWSS system provides for forms generation, data entry, class 
sampling, student sampling, student-teacher linkages, the random assignment of assessment booklets to 
students, the production of various survey tracking forms, and the printing of labels for test instruments 
and questionnaires. 

Westat home office staff attended a training session using WinWSS and conducted a 2-day training for 
the data entry staff to introduce them to the software and familiarize them with the listing form s and the 
procedures for entering data. 
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Obtaining Class Lists From Schools 


A Class Listing Form (CLF) was sent to the school coordinator. The CLF was used to create a list of the 
eligible classes, some attributes of each class, and the names of the teacher(s) teaching each class. An 
example Class Listing Form for grade 4 is reproduced in exhibit 4-1. A similar form for grade 8 is shown 
in exhibit 4-2. 

The information relating to school ID and school name, the name of the school coordinator, and the grade 
level in question was fdled before the form was dispatched. Schools were asked to complete the 
remaining information for each eligible fourth-grade class or eighth-grade mathematics class in the 
school. At grade 4, the form asked for the reading, math, and science teachers to be listed in addition to 
the class information. In grade 8, only the mathematics teachers were listed for each class. 

The CLFs were sent via FedEx in a packet to each designated school coordinator and contained a 
customized cover letter, the school’s CLF, and a school coordinator handbook for TIMSS-PIRLS or 
TIMSS (based on the grade). 

This packet was followed by an email with an encrypted CLF attachment that was password protected. 
Emails were sent to schools for which email addresses were available. All schools received a hard copy. 
The first wave of emails was sent January 27, 2011 (grade 4 CEFs); the second wave was sent February 1, 
2011 (grade 8 CEFs). 

Processing the Class Listing Forms. Receipt of the CLF was tracked using a ledger that included the 
return status of each form and detailed any anomalies. The status of schools was reviewed weekly. As 
CLFs were returned, they were processed as follows: 

■ The form s were reviewed for clarity and completeness. 

■ Any incomplete form s generated calls to the schools in question to provide the 
information or clarification necessary. 

■ Completed form s were passed to data entry staff where the information was entered 
into WinWSS, and a CLF in WinWSS system format was generated. 

■ The WinWSS CLF was compared with the original form as a check on the accuracy of 
data entry. 
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Exhibit 4-1. Example of TIMSS and FIRES fourth-grade class listing form with mock data 


T & P 11 - Class Listing Form 


T&P Participant Country 
School Name 

USA 

St. Peter Catholic School 

School ID 
School Coordinator Name 
Phone Number 

'3012 

Sister Mary Francis 

878-524-2354 


oeeoe © o © 


Class Name 

Grade 

Stream 

Number 

of 

Students 

Class 

Exclusion 

Status 

Name of Class Teacher or 
Reading / Language 
Teacher 

Name of Mathematics 
Teacher 

Name of Science Teacher 

41 

' 4 

' 0 

^15 


Sheri ta Bitten 

Adrian Alexander 

Adrian Alexander 

41 

’ 4 

' 0 

' 10 


Amanda Druckett 

Cynthia Monroe 

Cynthia Monroe 

4R 

' 4 

' 0 

’ 12 


Lisa Ki dwell 

Kristi Brooks 

Kristi Brooks 

4R 

' 4 

' 0 

' 12 


Janise Burton 

lennifer Ward 

Jennifer Ward 


































Grade (column 2) : 

Enter the grade for students In the class. Or. if the class is a mixed grade class, enter "M." 

Stream (column 3): 

Class streams occur when students are assigned to classes based on their ability. If the class is grouped by ability, please indicate so with the following codes: 0=not grouped: 1=low ability: 
2=average ability: 3= high ability. 

Class Exclusion Status (column 5): 

As a rule, all classes are to be included Examples of class-level exclusions include classes where all students belong to at least one of the following three exclusion status categories . 1 = 
students with functional disabilities: 2 = students with intellectual disabilities. 3 = non-native language speakers. If all students in the excluded class do not belong to the same exclusion 
category, please identify the category corresponding to the majority of students. All class-level exclusions must be approved 
Name of Reading/Language Teacher (column 6): 

Name of the teacher teaching reading/languge arts content to the class. 

Name of Mathematics Teacher (column 7): 

Name of the teacher teaching mathematics content to the class. 

Name of Science Teacher (column 8): 

Name of the teacher teaching science content to the class. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in International Mathematics 
^ and Science Study (TIMSS), 2011; Progress in International Reading Literacy Study (PIRLS), 2011. 
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Exhibit 4-2. Example of TIMSS eighth-grade class listing form with mock data 

T 11 - Class Listing Form - Grade 8 

TIMSS Participant Country usa 

School Name First Middle School 
School ID 3002 

School Coordinator Name Brenda croomes 
Phone Number 587-727-1258 


o e e o e © 


Class Name 

Grade 

Stream 

Number 

of 

Students 

Class 

Exclusion 

Status 

Name of Hathematics Teacher 

Algebra 8 A 

' 8 

3 

26 


Samantha Lewin 

Algebra 8B 

' S 

3 

26 


Samantha Lewin 

Pre-Algebra 8A 

' 8 

2 

32 


Mark Haygood 

Pre-Algebra 8B 

' S 

2 

33 


David Wellington 

Geometry 

M 

3 

10 


David Wellington 

General Math 1 

' 8 

0 

33 


Mark Haygood 

General Math 2 

' 8 

0 

32 


Samantha Lewin 

General Math 3 

’ 8 

0 

32 


Mark Haygood 




















Grade (column 2) : 

Enter the grade for students in the class. Or, if the class is a mixed grade class, enter "M.” 

Stream (column 3): 

Class streams occur when students are assigned to classes based on their ability. If the class is grouped by ability, please indicate so with the following codes: 0=not grouped: 1=low ability; 
2=average ability; 3= high ability. 

Class Exclusion Status (column 5): 

As a rule, all classes are to be included. Examples of class-level exclusions include classes where all students belong to at least one of the following three exclusion status categories: 1 = 
students with functional disabilities: 2 = students with intellectual disabilities: 3 = non-native language speakers, if all students in the excluded class do not belong to the same exclusion category, 
please identify the category corresponding to the majority of students. All class-level exclusions must be approved by the national center. 


SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in International Mathematics 
and Science Study (TIMSS), 2011. 


On 








Sampling Classes. WinWSS generated an equal probability sample of two classes (or pseudoclasses) 
where possible, based on the information in the CLF. If only one grade-appropriate class was available in 
a school, this class was selected with certainty. At grade 4, up to four classes were sampled and assigned 
to either TIMSS or PIRLS. The WinWSS sampling software handled the assignment of classes by 
randomly assigning the first sampled class and alternating assignment of each subsequent sampled class. 

In grade 8 schools, two classes were sampled unless the school only had one grade 8 class. The TIMSS- 
NAEP linking study at grade 8 required that one of the sampled classes be designated as the TIMSS- 
NAEP link class in order to administer a braided booklet of TIMSS and NAEP items. Once the classes 
were selected, a random number table (0-9) was used to define the TIMSS-NAEP link class. If the 
number was greater than or equal to 5, the class was designated as the link class. If the number was less 
than 5, the class was designated as the TIMSS class. The second class was assigned based on the 
designation of the first class. Once the link class was chosen, “LK” was added to the end of the class 
name to signify that the class was the link class. 


Identifying Students and Their Teachers 

With the classes identified, a Student-Teacher Linkage Form (STEF) was generated for each of the 
sampled classes in TIMSS and a Student Listing Form (SLF) was created for sampled classes in PIRLS. 
These form s were created by the WinWSS software and output in Excel using information from the CLF. 
An example of an STEF and SLF containing fictitious information is provided in exhibits 4-3 and 4-4, for 
TIMSS and PIRLS respectively. 

The upper part of the form containing the school ID, school name, class ID, school coordinator name, and 
grade was completed using information from the Class Listing Form. The remainder of the form was to be 
completed by the school, beginning with a listing of the names of the students in the named class. Schools 
were then asked to work across the form line-by-line to add the information relating to each student and to 
the mathematics and science teacher(s) of each student. 

Copies of the form s describing each of the sampled classes, along with instructions and examples, were 
sent to each school coordinator. 
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Exhibit 4-3. Example of TIMSS eighth-grade student-teacher linkage form with mock data 


T 11 - 

Student-Teacher Linkage Form - Grade 8 


School Name: Participant Country: 


ONE MIDDLE SCHOOL 


e 


United 

States 







□ 


[a] 

[b] 

[C] 

[d] 




ID 

Class ID 

class Name 

Grade 


■ 3001 r 300102 

Pre-Algebra P.3 


o 


e 

o 



€ 

> 



( 

9 

Student Name 

Date of Birth 

o 

TJ 

c 

(elusion Status 

Teacher Information 


0 Science Teacher 

Patricia Renz 
Pre-Algebra P. 3 
1 

Manny Foy 
General Science P.l 
6 

stormy Young 
General Science P. 5 
6 

stormy Young 
General Science P. 6 
6 

Cheri Young 
General Science P. 2 
6 

cheri Young 
General Science P.6 
6 

sacher's Name: 
ass (Course) Name: 
jbject Code: 

aacher's Name: 
ass (Course) Name: 
ibject Code: 

aacher's Name: 
ass (Course) Name: 
jbject Code: 

aacher's Name: 
ass (Course) Name: 
ibject Code: 

aacher's Name: 
ass (Course) Name: 
jbject Code: 

aacher's Name: 
ass (Course) Name: 
jbject Code: 

D Mathematics Teache 

or Number 

MM 

DD 

YYYY 

O 

Ul 

K U (0 

K U M 

HUM 

HUM 

HUM 

HUM 

2 

2 

Abrahams, Jeff 

1 

25 

1997 

2 


X 

X 







Adams, Joyce 

3 

14 

1997 

1 


X 





X 



Bailey, Kelli 

9 

6 

1997 

1 


X 




X 




Baxter, Lucas 

11 

7 

1997 

2 


X 


X 






Cuffman, Gage 

8 

2 

1997 

2 


X 



X 





Davies, Wallace 

6 

23 

1996 

2 


X 

X 







Denton, Patricia 

11 

15 

1996 

1 


X 

X 







Forth, Trent 

10 

11 

1997 

2 


X 



X 





Grant, Jackson 

12 

20 

1996 

2 


X 

X 







High, Teresa 

7 

6 

1997 

1 


X 

X 







Jefferson, Rudy 

4 

1 

1997 

1 


X 




X 




Jewels, Ashley 

2 

9 

1997 

2 


X 


X 






Kuben, Shannon 

1 

30 

1997 

2 


X 



X 





Lori ns, Scott 

6 

16 

1997 

1 


X 



X 





Luther, Mary 

9 

19 

1997 

1 


X 


X 






Maxwell, Stacey 

10 

10 

1996 

1 


X 




X 




Pace, Hilda 

12 

21 

1995 

1 


X 

X 







Page, John 

8 

31 

1997 

2 


X 



X ' 





Russell, A.J. 

9 

8 

1997 

2 


X 





X 



Rutx, Patrick 

5 

7 

1997 

2 


X 



X 





Sanchez, Paul 

7 

2 

1997 

2 


X 

X 







Simonton, Sia 

8 

3 

1997 

1 


X 





X 



Willmon, Lindsay 

4 

16 

1997 

1 


X 



X 












































































Gender (column 3): 1 = Female; 2 = Male 

Exclusion Status (column 4): 1 = Students with functional disabilities: 

2 = Students with intellectual disabilities; 3 = Non-native language speakers 
Subject Codes (Column 5): 1 = Mathematics; 6 = Integrated Science 

SOURCE: International Association for the Evaluation of Educational Achievement 
(lEA), Trends in International Mathematics and Science Study (TIMSS), 2011. 
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Exhibit 4-4. Example of PIRLS student listing form with moek data 


P 11 - Student Listing Form 


P Participant Country 

USA 

School Name 

St. Peter Catholic School 

School ID 

r 

3012 

Class Name 

4R-A 

Teacher Name 

Lisa Kidwell 


o e e o 


student Name 

Date of 
Birth 

Gender 

Exclusion 


DD 

YYYY 

Status 

Trey Judge 

r02 

14 

2000 

2 


Mark Beaufort 

t 03 

30 

2000 

2 


Cristin Viceory 

r 04 

01 

2000 

1 


Callie Frederick 

r 05 

15 

2000 

1 


Wendy Hill 

■07 

09 

2000 

1 


Frederick Beau 

^ 08 

02 

2000 

2 


Halford Gregory 

r 09 

16 

2000 

2 


Marcus Hughes 

r 10 

02 

2000 

2 


Valerie Sanders 

^ 11 

22 

1999 

1 


Elizabeth Lewis 

^11 

28 

1999 

1 


Suzanne Sutton 

^12 

05 

1999 

1 


Shelly Vandemeyer 

: 12 

30 

1999 

2 

































Gender (column 3): 

1 = Female: 2 = Male 
Exclusion Status (column 4): 

1 = Students with functional disabilities: 2 = Students with intellectual disabilities: 3 = Non-native language speakers 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 
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The response by schools to completing this somewhat formidable task was encouraging, although it 
required more follow-up than with the simpler Class Listing Form. The return of the STLF form s was 
tracked and reviewed weekly with the view to initiating reminder calls from field staff to the schools. 


Student Tracking Forms (STF) 

Information from the STLF was entered into the WinWSS system, which generated a Student Tracking 
Form (STF) for each class. The Student Tracking Form was designed to provide test administrators with 
student IDs, student identifiers in the form of date of birth and sex, the booklet assignment to each 
student, and the means to record the completion of the assessment and associated questionnaire. The 
student names or IDs shown in the first column of the form were removed following the assessment and 
retained by the school to ensure confidentiality. An example of a STF containing fictitious information is 
provided in exhibit 4-5. 


Teacher Tracking Form (TTF) 

A Teacher Tracking Form was also generated from information provided on the STLF. This form enabled 
test administrators to record the participation of teachers and to ensure that each teacher received the 
correct teacher questionnaire. Teacher names provided in the TTF were removed following the 
assessment and retained by the school to ensure confidentiality. An example of a TTF containing 
fictitious information is provided in exhibit 4-6. 
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Exhibit 4-5. Example of TIMSS eighth-grade student tracking form with mock data 


T 11 - Student Tracking Form - Grade 8 


School Name; TZMSS Participant Country: 


First Middle School 


United States 



[a] [b] [c] [d] [e] 


School ID 

Class ID 

class 

Name 

Grade 

Language 
of Test 

0453 

045307 

General 
Math 2 

8 

Engl i sh 

o 

e 

e o e o e 



Participation Status 



Main 

Session 

Hake-Up Session 

student Name 
or Number 

Student 

ID 

MM 

a 

a 

Date of Birth 

YYYY 

Gender 

Exclusion Status 

TIMSS Booklet 

Achievement Session 

Questionnaire Session 

Achievement Session 

Questionnaire Session 

Boyette, Dori 

04530701 

02 

13 

1997 

1 


1 





Bren ton, Paige 

' 04530702 

^03 

07 

1997 

' 1 


2 





Calzada, Kimberley 

' 04530703 

'05 

01 

1997 

' 1 


'3 





Cassano, Darrick 

^ 04530704 

'os 

24 

1997 

' 2 


4 





Cassano, Hector 

04530705 

'09 

08 

1997 

' 2 


5 





Cayer, Gordon 

04530706 

07 

11 

1997 

2 


6 





Danko, Shane 

04530707 

05 

09 

1997 

2 


7 





Doner, Christa 

04530708 

06 

14 

1997 

1 


8 





Ellwood, Ingrid 

04530709 

11 

19 

1996 

1 


9 





Esteban, Glenna 

04530710 

08 

17 

1997 

1 


10 





Fabrizio, Alta 

04530711 

10 

26 

1996 

1 


11 





Fleagle, Elvia 

04530712 

01 

31 

1997 

1 







Goodenough, lolene 

04530713 

02 

14 

1997 

1 


13 





He, Vernon 

04530714 

04 

07 

1997 

2 


14 





Holguin, Greg 

04530715 

07 

22 

1997 

2 


1 





Klick, Stefanie 

04530716 

12 

16 

1996 

1 


2 





Lever, Sam 

04530717 

11 

07 

1996 

2 


3 





Machin, Dustin 

04530718 

01 

06 

1997 

2 


4 





Magwood, Elisabeth 

04530719 

09 

09 

1997 

1 


5 





Olmedo, Corey 

04530720 

04 

23 

1997 

2 


6 





Piggford, Betsie 

04530721 

05 

19 

1997 

1 


7 





Santangelo, Clyde 

04530722 

02 

21 

1997 

2 


8 





Segovia, Derrick 

04530723 

08 

02 

1997 

2 


9 





Singley, Lewis 

04530724 

11 

27 

1996 

2 


10 





Spry, Rick 

04530725 

12 

28 

1996 

2 


11 





Studivant, Imogene 

04530726 

08 

25 

1997 

1 


12 





Ta rnowski , Dean 

04530727 

06 

01 

1997 

2 


13 





Thorpe, Glenda 

^ 04530728 


24 

’l997 

'' 1 


14 





Trump, Clyde 

r 04530729 


11 

'1997 

'' 2 


1 





Ugalde, Rachelle 

r 04530730 


23 

'l997 

'' 1 


2 ! 





Westbury, Wesley 

^04530731 


09 

F 997 

r-j— 


1 

3 





Vallo, Herman 

^ 04530732 

IF 

29 

'l996 

'' 2 


f 

4 





<A> 

04530733 






5 





<A> 

04530734 






6 





<4> 

04530735 






7 






Gender (column 4): 1 = Female; 2 = Male 

Exclusion Status (column 5): 1 = Students with functional disabilities: 

2 = Students with intellectual disabilities: 3 = Non-native language speakers 

Participation Status (column 7): C = Participated;A = Absent; P = No parental permission: NA = Left school 
permanently 

SOURCE: International Association for the Evaluation of Educational Achievement 
(lEA), Trends in International Mathematics and Science Study (TIMSS), 2011. 
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Exhibit 4-6. Example of TIMSS eighth-grade teacher tracking form with mock data 


T 11 - Teacher Tracking Form - Grade 8 


School Name: Participant Country: 


First Middle School 


United States 



[a] [b] 




School ID 

Grade 


o 

' 0453 


e 

e o 

0 © O © 

Teacher Name 

Teacher ID 

Link Number 

Selected Class 
ID 

Class (Course) 
Name 

Number of 
Eligible 
Students 

Subject and 
Questionnaire 
Code 

Questionnaire 

Return 

Status 

Samantha Lewin 

'' 045301 

' 01 

045302 

Algebra 8B LK 

' 26 

1 

0 

Samantha Lewin 

w 

045301 

W 

02 

w 

045307 

General Math 
2 

r 

32 

f 

1 

0 

Bob Knight 

045302 

f 

03 

f 

045302 

Advanced 
Science 8 

f 

7 

f 

6 

0 

Ginny Buchanan 

r 

045303 

f 

04 

w 

04302,045307 

General 
Science 8 

r 

31 

f 

6 

0 

Marsha Howe 

r 

045304 

w 

05 

'045302,04530 

7 

General 
Science 8 

r 

20 

f 

6 

0 










































Subject and Questionnaire Codes (column 7): 

1 = Mathematics: 6 = Integrated Science: 

Questionnaire Return Status (column 8): 

N = Not completed: R = Returned completed: 0 = To be completed online 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in International Mathematics and Science Study (TIMSS), 
2011 . 


o 

INO 




4.3 


Instruments for TIMSS and PIRLS 


All TIMSS and PIRLS instruments were developed by the lEA as a cooperative effort involving 
representatives from every country participating in the study. For TIMSS, at each grade the primary 
instruments for TIMSS 2011 consisted of a combined mathematics and science assessment; a student 
questionnaire; a curriculum questionnaire; a school questionnaire; and teacher questionnaires to be 
completed by the teachers teaching mathematics and/or science to the students in the sampled classrooms. 
For PIRLS the instruments included a reading assessment; a student questionnaire; a curriculum 
questionnaire; a school questionnaire; and teacher questionnaires to be completed by the teachers teaching 
reading to the students in the sampled classrooms. 

The TIMSS assessment instrument took the form of 14 booklets containing mathematics and science 
items in rotated item blocks. While some of these items have to remain confidential for use in coming 
TIMSS assessments, others have been released as actual examples of assessment items. TIMSS 2011 
items released by the lEA are available on the TIMSS & PIRLS International Study Center website at 
http://timssandpirls.bc.edu/timss201 1/intemational-released-items.html and also the NCES TIMSS 
website at http://nces.ed.gov/timss/educators.asp . 

The assessment for PIRLS consisted of 13 booklets and presented items in rotated item blocks. Like 
TIMSS, some items have to remain confidential for use in coming PIRLS assessments while others have 
been released as examples of the assessment. PIRLS 2011 items released by the lEA are available on the 
TIMSS & PIRLS International Study Center website at 

http://timssandpirls.bc.edu/pirls201 1/international-released-items.html and also the NCES PIRLS website 
at http://nces.ed.gov/surveys/pirls/released.asp . 

The student questionnaire was grade-specific and bound into the assessment booklet with the assessment 
items for the particular grade. The school questionnaire was designed as an online instrument to be 
completed by the school principal. In the case of the teacher questionnaires, these were also developed as 
online instruments. A single fourth-grade online teacher questionnaire was designed for both TIMSS and 
PIRLS to be completed by the reading and/or mathematics and/or science teachers of the students in the 
sampled classrooms. Separate questionnaires were provided for the TIMSS mathematics teachers and 
science teachers of the students in the sampled eighth-grade classrooms. The international versions of the 
TIMSS questionnaires are available on the TIMSS & PIRLS International Study Center website at 
http://timssandpirls.bc.edu/timss201 1/intemational-contextual-q.html . The international versions of the 
PIRLS questionnaires are available on the TIMSS & PIRLS International Study Center website at 
http://timssandpirls.bc.edu/pirls201 1/intemational-contextual-q.html . The U.S. versions of all seven U.S. 
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TIMSS questionnaires and the three PIRLS questionnaires are provided in appendix F of this report and 
are available on the NCES website at http://nees.ed.gov/timss/questionnaire.asp and 
http://nces.ed.gov/survevs/pirls/questionnaire.asp respectively. 


The TIMSS Mathematics and Science Assessment and PIRLS Reading 
Assessment 

The following discussion provides a summary of the rationale for and development of the TIMSS and 
PIRLS 2011 assessments. Complete detail is provided in the Methods and Procedures in TIMSS and 
PIRLS 2011 (Martin & Mullis, 2011) available at http://timssandpirls.bc.edu/methods/index.html. 


Assessment Frame works 

For both TIMSS and PIRLS 2011, the test development effort began with a revision of the frameworks 
used to guide the construction of the previous TIMSS 2007 and PIRLS 2006 assessments (Mullis et ah, 
2005; Mullis et ah, 2006). The frameworks were updated to reflect changes in the curriculum and 
instruction of participating countries (Mullis et ah, 2009). Extensive input from experts in mathematics 
and science education, reading education, assessment, and curriculum, and representatives from national 
educational centers around the world contributed to the final shape of the frameworks. Maintaining the 
ability to measure change over time was an important factor in revising the frameworks. 


Content and Cognitive Domains 

The TIMSS mathematics and science assessments were designed along two dimensions: (1) the topics or 
content that students are expected to learn and (2) the cognitive skills students are expected to have 
developed. In TIMSS 2011, the cognitive domains are knowing, applying, and reasoning and are common 
to each grade and subject area. The content areas differ between grades and subject areas. The 
mathematics topical or content domains covered at grade 4 are number, geometric shapes and measures, 
and data display. At grade 8, the mathematics content domains are number, algebra, geometry, and data 
and chance. The science content domains covered at grade 4 are life science, physical science, and Earth 
science. At grade 8, the science content domains are biology, chemistry, physics, and Earth science. 
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The PIRLS 2011 reading assessment design focused on two dimensions: (1) a content dimension 
specifying the purpose for reading and (2) a cognitive dimension specifying the cognitive or thinking 
processes of comprehension. The two content domains assessed in PIRLS, called “purposes of reading,” 
are literary experience and acquire and use information. PIRLS assessed students’ reading literacy in four 
cognitive areas, called “processes of comprehension”: focus on and retrieve explicitly stated information; 
make straightforward inferences; interpret and integrate ideas and information; and examine and 
evaluate content, language, and textual elements. 


Item Development 

Approximately one -half of the assessment items used in TIMSS 2007 and in PIRLS 2006 were kept 
confidential and included in the respective TIMSS and PIRLS 2011 assessments. To replace assessment 
items that had been released following the 2007 and 2006 assessments, education systems submitted 
items for review by subject-matter specialists, and additional items were written by the lEA Science and 
Mathematics Review Committee for TIMSS and the lEA Reading Review Committee for PIRES, in 
consultation with item-writing specialists. This expert consultation ensured that the content, as explicated 
in the frameworks, was covered adequately. Items were reviewed by the Science and Mathematics Item 
Review Committee and the Reading Item Review Committee and field-tested in most of the participating 
education systems. Results from the field test were used to evaluate item difficulty, how well items 
discriminated between high- and low-performing students, the effectiveness of distracters in multiple- 
choice items, scoring suitability and reliability for constructed-response items, and evidence of bias 
toward or against individual countries or in favor of boys or girls. 

As a result of this review in TIMSS, 144 new fourth-grade items were selected for inclusion in the 
international assessment. In total, 347 mathematics and science items were included in the fourth-grade 
TIMSS assessment booklets. At the eighth grade, the review of the item statistics from the field test led to 
the inclusion of 183 new eighth-grade items in the assessment. In total, 434 mathematics and science 
items were included in the eighth-grade TIMSS assessment booklets. In PIRES, 60 new reading items 
were selected for inclusion in the international assessment. In total, 135 reading items were included in 
the PIRES assessment booklets. More detail on the distribution of new and trend items in the TIMSS and 
PIRES 2011 assessments can be found in Martin and Mullis (201 1). 
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Assessment Booklets 


To keep the testing burden to a minimum while ensuring broad subject-matter coverage, TIMSS used a 
rotated block design that included both mathematics and science items. This is consistent with other large- 
scale assessments such as the National Assessment of Educational Progress (NAEP). The rotated item 
design meant that, while no students responded to all of the items, each student encountered both 
mathematics and science items during the assessment. 

The 2011 fourth-grade assessment consisted of 14 booklets, each requiring approximately 72 minutes of 
response time. In the TIMSS 2011 booklets the mathematics and science items were each assembled 
separately into 14 blocks, or clusters, of items. Each block contained either mathematics items or science 
items only. The secure trend items used in prior assessments were included in eight blocks, with the other 
six blocks containing new items. Each of the 14 TIMSS 2011 booklets contained four blocks in total. 

The 2011 eighth-grade assessment followed the same pattern and consisted of 14 booklets, each requiring 
approximately 90 minutes of response time. The 14 booklets were rotated among students, with each 
participating student completing 1 booklet only. The mathematics and science items for TIMSS 2011 
were assembled into 14 blocks of items. Each block contained either mathematics items or science items 
only. The secure trend items used in prior assessments were included in eight blocks, with the other six 
blocks containing new items. Each of the 14 TIMSS 2011 booklets contained four blocks in total. The 
TIMSS booklets administered to the state samples were identical to those administered in the national 
sample. 

The PIRES 2011 assessment was similar in design to TIMSS. The PIPES assessment consisted of 12 
booklets and a reader that was presented in a magazine -type format, with the questions in a separate 
booklet. Each booklet required 80 minutes of response time. The reading items were assembled separately 
into 10 blocks, or clusters, of items. Each of the 13 booklets contained one block of literary experience 
items and one block of informational items, and each block occurred twice across the 13 booklets. The 13 
booklets were rotated among students with each participating student responding to only 1 booklet. Six of 
the 10 blocks were included in previous PIPES assessments. The remaining 4 blocks were new for PIRES 
2011 . 
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Calculator Use in TIMSS 


Calculator use was not permitted during the TIMSS fourth-grade assessment. However, the TIMSS policy 
on calculator use at the eighth grade was to give students the best opportunity to operate in settings that 
mirrored their classroom experiences. As a consequence, calculators were permitted, but not required, for 
the eighth-grade assessment materials. In the United States this meant that students assigned one of the 14 
TIMSS 2011 booklets were allowed, but not required, to use calculators. 


Background Questionnaires 

As in prior administrations, TIMSS and PIRLS 2011 included self-administered questionnaires for 
principals, teachers, and students. To create the questionnaires for 2011, the 2007 versions of the TIMSS 
questionnaires and the 2006 versions of the PIRLS questionnaires were reviewed extensively by the 
national research coordinators from the participating countries as well as a Questionnaire Item Review 
Committee (QIRC). Based on this review, the QIRC eliminated or revised some questions and added 
several new ones. Like the assessment items, all questionnaire items were field-tested and the results 
reviewed. As a consequence, some of the questionnaire items were revised prior to their inclusion in the 
final questionnaires. The questionnaires requested information to help provide a context for the 
performance scores, focusing on such topics as students’ attitudes and beliefs about learning, their habits 
and homework, and their lives both in and outside of school; teachers’ attitudes and beliefs about teaching 
and learning, teaching assignments, class size and organization, instructional practices, and participation 
in professional development activities; and principals’ viewpoints on policy and budget responsibilities, 
curriculum and instruction issues, and student behavior, as well as descriptions of the organization of 
schools and courses. For 2011, online versions of the school and teacher questionnaires were offered to 
respondents as the primary mode of data collection. Paper versions were also available. Detailed results 
from the student, teacher, and school questionnaires are available in the two TIMSS international reports: 
the TIMSS 2011 International Mathematics Report (Mullis et ah, 2012) and TIMSS 2011 International 
Science Report (Martin et ah, 2012); and in PIRLS 2011 International Report in Reading (Mullis et al. 
2012). 

U.S. Adaptations to the Assessment Items and Questionnaires 

Source versions of all instruments (assessment booklets, questionnaires, and manuals) were prepared by 
the lEA in English and translated by the participating countries into the primary language or languages of 
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instruction in each country. In addition, it was sometimes necessary to adapt the instruments to better fit 
language usage, even in countries that use English as the primary language of instruction. Other 
adaptations to fit national education characteristics were sometimes required as well. All adaptations were 
reviewed and approved by the lEA to ensure they did not change the substance or intent of the question or 
answer choices. 


U.S. Adaptations to the Assessment Items 

As in previous cycles of TIMSS and PIRLS, the U.S. adaptations to the international instruments were 
minimal and designed to make the assessment more readable to U.S. students without changing the 
essence of the assessment item. For example, at times names of individuals were changed to more 
familiar form s (for example, “Ahmed” to “Andrew”), nouns with British origins were changed to their 
U.S. equivalent (for example, “cinema” to “movie theater”). Imperial English spellings were changed to 
American English (for example, “organisation” to “organization, “programme” to “program”), and some 
changes to the text of instructions were also made to better mirror the administration procedures in the 
United States. 


U.S. Adaptations to the Sehooi, Teaeher, and Student Questionnaires 


Adaptations made to the school, teacher, and student questionnaires were of the following five main 
types: 


■ changes to general instructions made in the interests of enhancing clarity; 

■ changes designed to make question text more readable to U.S. students, similar to 
those made to the assessment items as described above; 

■ changes to response alternatives where the international response set did not 
adequately reflect the U.S. context; 

■ additional questionnaire items included to address particular issues of national 
interest; and 

■ International items that were omitted from the U.S. questionnaires because they 
violated the federal Pupil Privacy Rights Act (for example, questions on bullying and 
violence in the school). 
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A detailed list of changes made to the questionnaires is provided in appendix G Both the original text 
from the international version of the questionnaire and the changed text from the U.S. version are shown. 
Text that has been changed in the U.S. version is underlined in that version. Both international and U.S. 
questionnaire item numbers, or other location indicators, are provided in each instance. Where 
appropriate, a crosswalk between the U.S. and international versions of the set of response categories of 
items is provided in the “Comments” column. 


Translation and Verification of Instruments 

Each country prepared translations of the instruments according to translation guidelines established by 
the lEA. Since the international versions of the instruments are produced in English, the U.S. did not need 
to engage in the full-fledged translation required of many nations. However, the adaptations made to the 
U.S. instruments required verification by the lEA to ensure their suitability for the current cycle of 
TIMSS and, if trend items, their continuity with previous cycles. Further details on the translation process 
can be found in Martin and Mullis (2011). 


Production of Assessment Booklets and Questionnaires 

On receiving lEA approval of the adaptations, Pearson Educational Measurement applied the adaptations 
to the international questionnaires and item blocks and then assembled the final assessment booklets. 
Quality control procedures for this process included a review of each adaptation made to the 
questionnaires and item blocks as well as a full review of the assembled instruments in a final layout 
proof 

In mid-December 20 1 0, electronic files were sent to the lEA Data Processing Center (DPC) for 
verification of the national changes and to the TIMSS & PIRES International Study Center for layout 
verifications. The student assessment booklets and student questionnaires were printed in scannable form. 
The teacher and school paper and pencil versions of the questionnaires were printed in non-scannable 
form. 

A similar procedure was applied to the online school and teacher questionnaires. Adaptations were made 
to the online instruments using software provided by the lEA DPC. A set of output files was produced for 
each questionnaire, and these were uploaded to a secure sever and verified by the DPC. 
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Scannable Document Preparation and Printing 


The student test assessment and student questionnaire were printed as scannable documents combined 
into 14 single booklets. Document production was divided into two phases. The preparation phase 
included the mockup and design of forms, typesetting, composition, and editing for text accuracy and 
process ability. The production phase included final platemaking, printing, binding, and any finishing 
procedures (counting, wrapping, etc.) that were required prior to packaging and distribution. 

After a form was created, it was thoroughly inspected for grammar, spelling, and punctuation to ensure 
that it matched the approved electronic files. Subsequently, copies of these proofs were subject to a 
technical review of “scan ability,” oval placement, and spine code assignment. Immediately after printing, 
sample documents were selected from predetermined locations throughout the print run for testing. Before 
shipping, a sample from each carton of multipage documents was inspected to ensure that the pages of the 
booklets were in the correct sequence. After binding, all documents were boxed to assure that material 
quality was maintained during transit. 


Preparation and Printing of Non-Scannabie Documents 

The teacher and school questionnaires were produced as non-scannable documents. In a first stage proofs 
of each document were reviewed against the original electronic files. Once accuracy was certified, 
printing was initiated. During this process staff checked a 1 0 percent sample of the printed form against 
the approved document to ensure that accuracy was maintained throughout the printing process. 


Oniine Questionnaires 

Once verified, each questionnaire was loaded to an NCES server and thoroughly tested by NCES and 
Westat to ensure that responses were being captured correctly and that the instruments were functioning 
properly. 


4.4 Field Operations 

The activities discussed here refer to those associated with the administration of the assessments in 
participating schools. In the United States the administration of the assessments was carried out by 
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professional staff trained according to the international guidelines. School personnel were asked only to 
assist with listings of students, the identification of school space for the assessment, and the specification 
of parental consent procedures needed for sampled students. 

Field operations centered on three main tasks: recruiting and training of field staff; scheduling the 
assessments; and administration of the assessments within the schools. 

Many of the gaining cooperation staff working on the recruitment of districts and schools were retained 
for the assessment phase of the study. However, it was necessary to recruit additional staff at the test 
administrator’s level from the pool of experienced Westat field staff. All had previous experience with 
other educational assessments in schools, and all had FBI clearance based on fingerprint and background 
checks. Test administrators also signed a statement of nondisclosure indicating that they would maintain 
confidentiality of all survey materials and of the data collected. The test administrators recruited local 
assistant administrators to assist in the administration of the assessment since in most schools two 
separate classrooms were assessed simultaneously. Assistant administrators were also experienced with 
the administration of educational assessments in schools and held the same kind of FBI clearance. They 
too signed confidentiality statements. 

The total complement of field staff consisted of four field managers, 135 test administrators, and 135 
assistant administrators. The field managers reported directly to the Westat home office and met weekly 
to discuss progress and any problematic issues arising in the field. Test administrators reported to a field 
manager who coordinated and monitored their work and, in turn, the test administrators coordinated and 
supervised the work of the assistant administrators. 


Responsibilities of Field Managers, Test Administrators and Assistant 
Administrators 

Field managers involved in the data collection phase had responsibility for 

■ providing input to the TIMSS and PIRLS data collection training; 

■ tracking the test administrators’ receipt of assessment booklets and other materials; 

■ coordinating data collection activities undertaken by their assigned test administrators; 

■ holding weekly one-on-one telephone meetings with their test administrators to 
monitor progress and to troubleshoot any problems arising; 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


111 



ensuring that their test administrators followed TIMSS and PIRLS procedures and 
guidelines; and 


■ reporting progress and problems in weekly conference calls with Westat home office 
staff and other field managers. 

Test administrators had responsibility for 

■ attending TIMSS and PIRLS data collection training; 

■ receiving and securing assessment materials; 

■ training assistant administrators; 

■ preparing and assigning assessment materials for students; 

■ collecting the completed school and teacher questionnaires; 

■ conducting the assessment according to TIMSS and PIRLS specified procedures; 

■ completing the Test Administration Form, Student Tracking Form, Teacher Tracking 
Form, and Student Response Rate Form; 

■ determining if a follow-up session was needed; 

■ securing, packing, and shipping all assessment materials to Pearson at the conclusion 
of the assessment; 

■ recording the status of the assessment in the Field Management System; and 

■ reporting progress to their field manager on a regular basis. 

Assistant administrators had responsibility for 

■ attending the TIMSS and PIRLS training conducted by their test administrator; 

■ administering the assessment according to TIMSS and PIRLS specified procedures; 

■ completing the Test Administration Form, Student Tracking Form, Teacher Tracking 
Form, and Student Response Rate Form; and 

■ consulting regularly with their test administrator. 


Training 

A 2-day, in-person training for test administrators was held on March 1 8 and 19,2011. The attendees 
received the Test Administrator Manual 5 days prior to the training session and were given four paid 
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“study hours” to become familiar with the information prior to training. The agenda for this training 
session is provided as exhibit E-1 in appendix E. 

Since TIMSS and PIRLS were very similar in terms of general procedures, the training was centered on 
TIMSS eighth-grade. TIMSS at grade 4, the TIMSS-NAEP Eink, and PIRLS were referenced at specific 
times where a unique training point needed to be made or when procedures differed. The first day of 
training focused on the responsibilities of a TIMSS test administrator, TIMSS data collection materials 
from the perspective of item security and student privacy, the Preassessment Call Checklist, and entering 
preassessment call information into the FMS. The second day included discussion about the procedures to 
be followed in preparing for, arriving on, and conducting the assessment day, what to do once the 
assessment was completed, and the appropriate methods for packing and shipping the assessment 
materials to Pearson. Subsequently, the focus shifted to the PIRLS, TIMSS, and TIMSS-NAEP link 
administrative duties. At the end of day 2, test administrators received their initial assignments and met 
with their field managers to address questions and other administrative details. 

Field staff were assigned laptop computers to take with them for the duration of TIMSS and PIRLS data 
collection. Printers for use with the laptops were issued and sent to each test administrator after training. 
Test administrators were also provided with an official TIMSS 2011 photo ID badge to wear while 
representing TIMSS and PIRLS in the schools. 

Training for assistant administrators (AAs) was conducted as a separate exercise by their respective test 
administrator. Approximately 2 hours was allocated for this training. Training materials consisted of an 
Assistant Administrator Manual, session scripts, and a subset of material from the test administrator 
training. 


Scheduling Assessments 

The 1,240 schools taking part in TIMSS and PIRLS 2011 were dispersed across the country, although 
with concentrations in the more populous states and cities. Scheduling assessment dates for these schools 
required the optimization of school preferences for a particular date with the assessment date preferences 
of nearby schools and the location of field staff in an effort to keep travel and related expenses to a 
minimum. The basic approach adopted involved the geographic mapping of schools and assignment of a 
preliminary date, along with the location of field staff. This formed the foundation for discussions with 
schools, and the assignment of schools to field staff. In essence, geographical clusters of schools were 
assigned assessment dates clustered in time insofar as this was possible. 
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Obtaining Assessment Dates From Sehoois 


Prior to recruitment of schools, Westat staff collected date information for the sampled schools, including 
dates associated with state testing, holidays, breaks, field trips, and the like. This information was used to 
set a tentative date for each school. 


Mapping Sehoois 

In a second step a geographic map of participating grade 4 and grade 8 schools was developed based on 
the addresses of the schools. The map allowed the ready identification of obvious clusters of schools in 
large metropolitan areas and also allowed for identification of less concentrated clusters in broader 
geographical areas as well as isolated schools. To establish optimal work areas for assessment 
administrators, geographic clusters of schools were defined in an iterative process that took into account 
the location of field staff and their caseload. A total of 134 work areas of varying size were identified in 
this way. 


Assigning Tentative Assessment Dates 

The provisional assessment dates were then mapped to the schools in each cluster and represented in a 
spreadsheet. Within clusters, assessment dates were balanced against the location of schools in relation to 
one another. With some minor modifications to work areas, schools in relatively close proximity were 
assigned to provisional assessment weeks. This process identified smaller clusters of schools that could be 
assessed within a week without undue travel and related expenses. For the most part, assessments were 
scheduled on Tuesday through Thursday, leaving Monday and Friday as travel days. 


Negotiating Finai Assessment Dates 

Once tentative dates were established, field staff contacted schools to negotiate a final assessment date. 
For the most part, schools accepted the dates proposed. When this was not possible, field staff used the 
work area spreadsheet and a work area map to negotiate an alternative assessment date that would 
minimi z e staff travel. Final assessment dates and times were confirmed by email with the school 
coordinator. 
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Assignment of Schools to Test Administrators 


For the most part, field staff were assigned a work area based on their location and availability with most 
work areas relatively close to the test administrator’s home address. Balancing these several demands 
resulted in some variation in the caseload of test administrators. During the course of the assessments, 
which ran from March 2011 through to June 2011, some reassignments of schools and/or work areas were 
necessary. 


Troubleshooting 

Fifteen of the test administrators and 42 assistant administrators were recruited as regional 
troubleshooters who would be available to cover last-minute changes in assessment dates, staff illnesses, 
and the like. Each of these staff also had their own assignments, essentially schools that did not easily fit 
into an identified work area or assessment date window. 


The Assessment and Related Activities 

Field staff engaged in a number of activities in advance of the actual assessment. These included 

■ working with the school coordinator to gain the permission of parents and students, if 
this was required by the school; 

■ making arrangements with the school for the assessment sessions; and 

■ obtaining the materials to be used in the assessment. 


Recruiting Parents and Students 

During recruitment and scheduling contacts with schools, field staff asked about district and/or school 
requirements for notifying parents about their child’s participation in TIMSS. School requirements fell 
into the following three main categories: 

■ Notification, The school would simply send parents a notification of the child’s 
participation in TIMSS along with some informational material; 
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■ Passive consent. The school was required to ask parents for permission for the child 
to participate but permission would be assumed unless there was a formal objection; 
and 

■ Active consent. The school was required to ask parents for permission for the child to 
participate and the child could not participate until the parents provided formal 
approval. 


A majority of schools (956) opted for parent notification. Nearly 400 schools used passive consent and a 
further 50 used active consent. 


To assist schools in this task the school coordinator was provided with three draft letters to parents, one 
for each of the three form s of parent permission. These letters could be edited as appropriate and sent out 
on school stationery. Consent form s to accompany the passive and active consent letters were also 
provided along with an information sheet describing TIMSS. English and Spanish versions of each of 
these documents were made available to the schools. Copies of these materials as they apply to fourth- 
grade students and eighth-grade students are provided as exhibits C-6 through C-8 in appendix C. The 
eighth-grade materials were essentially the same. 


Organizing the Assessment Session at the Sehooi 

Approximately 2 weeks prior to each school’s assessment date, the test administrator called the assigned 
school coordinator. Using a Preassessment Call Log, test administrators were instructed to verify 
previously obtained information on items such as the school’s address, principal’s name, assessment date, 
session location, requirements for entering the school, and parking arrangements as well as the status of 
the within-school sampling form s for the school. The information obtained was maintained within the 
School Folder and updated as a basic reference. The Preassessment Call Log is provided as exhibit E-2 in 
appendix E. 

On assessment day each test administrator, accompanied by an assistant administrator, arrived at the 
school with all of the materials needed for the assessment. One session box of materials was provided for 
each of the sampled classes. Each session box contained the estimated number of student assessment 
booklets required, plus three unassigned booklets to accommodate any changes in class enrollments. 

Upon arrival, the test administrator met with the school coordinator to make any updates to the Student 
Tracking Form that would affect the preparation of student materials (for example, the addition of new 
students, the withdrawal of listed students from the school or class, or a change in exclusion status of a 
sampled student). 
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Test administrators arrived at the school about an hour before the scheduled assessment to prepare 
booklets for distribution and take care of other arrangements for the assessment. Following the prescribed 
international procedures, the test administrator did not open the booklet bundles until 45 minutes before 
the assessment. At that time, the booklets were assigned to students in the random order established by 
the lEA sampling software, and labels were placed on the booklets. TIMSS pencils were provided to all 
students. As required by international rules, simple-fimction calculators were provided at the eighth-grade 
level only, and then only at the discretion of the school. Students kept the TIMSS pencils as gifts, but 
calculators distributed during the assessments were collected with the booklets after the assessment. 


Administering the Assessment 

Assessments were administered by reading verbatim from a standardized script according to the 
instructions in the TIMSS & PIRLS Test Administrator Manual. A copy of each of the sessions scripts are 
provided in appendix E. The script began with a brief introduction to the study. The assessment booklets, 
each in a security envelope, then were distributed. The students were instructed to remove their booklet 
from the envelope, and the general instructions and instructions for Part 1 were read. Following this, the 
students were instructed to begin Part 1 of the assessment. After 36/45 minutes (grade 4/grade 8), a short 
break was provided. After the break, the instructions for Part 2 were read and students were instructed to 
begin Part 2 of the assessment. After the allotted 36/45 minutes for this part of the assessment, students 
were instructed to stop work, and a longer break was provided. Following the break, the student 
questionnaire was administered; it was not time -limited but was typically completed in about 30 minutes. 


Postassessment A cti vities 

Following the assessment, test administrators instructed the students remove an identifying label from the 
cover of the booklet, place the booklets in the security envelope, and seal it. The students handed their 
booklets to the administrator, received their gift, and were dismissed. The test administrator then recorded 
participation codes for each session and packed the booklets and school and teacher questionnaires into 
the shipping box. The test administrators made copies of the Student Tracking Form and the Session 
Report Form, placing the original documents in the TIMSS and/or PIRES Storage Envelope to be kept at 
the school. To maintain the security of student names, the test administrator removed and destroyed the 
column of student names from the copies of the STF and placed the de-identified STF and Session Report 
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Form in with the assessment booklets and questionnaires. The session materials were sealed and shipped 
to Pearson. 


4.5 Receipt Control, Scoring, Coding, and Data Entry 

As noted previously, field staff sent the completed assessments and questionnaires along with any related 
materials directly to Pearson following the completion of the assessment session at a school. Pearson then 
recorded the receipt of materials, scored the open-ended responses in the assessment, coded the multiple- 
choice assessment items and the questionnaire responses, and created data files from this information. 


Receipt Control 

TIMSS and PIRLS documents were received at Pearson from April 6 to June 3, 201 1. As assessment 
materials were returned to Pearson, receipt dates were recorded and sent to Westat weekly. Pearson also 
provided Westat with updated assessment information taken from the Student Tracking Forms. 

For TIMSS Pearson checked in and processed materials from approximately 370 grade 4 and 501 grade 8 
national schools. Pearson processed an additional 103 grade 4 and 395 grade 8 state schools in Florida 
and North Carolina that participated as benchmarking states. School personnel could complete their 
teacher and school questionnaires online or return them with the assessment booklets. Those returned to 
Pearson were entered by project staff into the NCES secure website. 

For PIRLS Pearson checked in and processed approximately 13,400 assessment booklets from 370 
schools in the national sample. Approximately 2,000 additional booklets from 57 schools in Florida were 
also processed. Again, school personnel could complete their teacher and school questionnaires online or 
return them with the assessment booklets. Those returned to Pearson were entered by project staff into the 
NCES secure website. 


Receipt Control System Specification 

Two systems were used to monitor the receipt and processing of materials after the assessments — the 
Process Control System and the Work Flow Management system. These systems enabled project staff to 
determine the status of any selected school; verify that materials from a completed school had been 
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received; identify discrepancies in student or school information; and obtain information on the status of 
data processing activities for a particular batch of materials. 

The Process Control System was modified to monitor the status of all sampled schools and test materials 
for each study. The Process Control System contained a list of all participating schools and the 
completion status of the schools for the TIMSS and PIRLS assessments. Shipments were returned 
packaged in their original boxes, with a barcoded label, applied prior to shipping the booklets out for the 
assessment, containing the school ID. The barcode was scanned upon arrival at Pearson. A receipt report 
was run each day and forwarded to the Westat home office on a weekly basis. 

When shipments were received staff checked each shipment to verify that the contents of the box matched 
the school indicated on the label. Each shipment was then checked for completeness and accuracy based 
on procedures outlined in the lEA Survey Operations Manual. Any discrepancies were recorded and 
project staff alerted to determine the cause. Once a shipment was opened and verified as being complete, 
the documents were organized into work units and batched. The computerized Work Flow Management 
System was used to track the documents through every processing step, thus enabling project staff to 
easily locate materials for a particular school. 


Booklet Accountability 

Prior to the distribution of materials, all assessment instruments were organized into bundles and scanned 
to a file that was used to control distribution to field staff or a particular school. This assignment was 
recorded in the Materials Distribution System. 

When the return shipments were received, a manual count was made to ensure that all booklets from the 
original bundle were there. The assessment booklets were submitted for scanning and key entry, and 
unused booklets were batched and the booklet identification barcode scanned. This file and the processed 
documents file were compared to the original bundle security file created before distribution. A list of 
unmatched booklet identification numbers was printed in a report used to confirm any nonreceipt of 
individual booklets. 

After the batches of documents had successfully passed the coding and editing process, they were sent to 
the warehouse for storage. The storage locations of all documents were recorded on the inventory control 
system, which permits the rapid retrieval of any document, should it be necessary. Unused materials were 
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sent to temporary storage until the study was completed and the data files accepted by the Westat home 
office, at which time the extra inventory was destroyed. 


Scoring the Assessment Items 

The TIMSS and PIRLS assessment items included both multiple -choice and constructed-response items. 
Scoring rubrics developed internationally following the field tests of the assessment items were available 
to guide the scoring of each constructed-response item. In the United States, the scoring of the open- 
ended student responses according to these rubrics was the responsibility of Pearson. 


Training 

Two subject-specific scoring directors participated in the TIMSS training sessions sponsored by lEA, and 
an additional scoring director participated in the PIRLS training sessions. Materials from these sessions 
along with additional materials constructed specifically for this purpose were used to train team leaders 
and supervisors. There were 10 scoring supervisors for TIMSS. All were hired based on their experience 
with similar mathematics and science scoring projects. Pearson hired 78 scorers for TIMSS organized into 
12 teams, with 6 to 8 scorers per team. For PIRLS, two scoring supervisors were hired along with 18 
scorers organized into two teams of 9. 

Training activities for the scorers followed the same routine, with supervisors leading each small team 
reading the item prompt; reading the rubric or scoring guide aloud; reading aloud each of the anchor 
papers and explaining the reasoning behind the score; allowing the scorers time to complete the practice 
papers; reviewing each of the practice papers; and opening individual scoring on the electronic 
Performance Evaluation Network (ePEN). Training on and scoring of the items was completed one at a 
time, with the exception of the linked items, which were scored together. 


Scoring 

Each team worked on different items. Scorers were able to view only the one active item that they were 
assigned by their supervisors and were not able to score other items from any other team. The exception 
was with the linked items from the “themed” blocks. In this situation a question or questions required 
reference to a previous answer. For instance, question 4 in the block may have required data or the answer 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


120 



from questions 1 and 2 in order to answer the question. In this situation, the ePEN setup was designed so 
that the scorer had access to screens with all of the pertinent responses, again only as assigned by the 
supervisor. The answers could not be submitted to ePEN until all questions from the set were scored to 
make sure that the scorer had all the needed information to score all related items. 

Scoring quality was monitored continuously. The ePEN system allowed inter-rater reliability reports to be 
run almost as soon as scoring began. Another monitoring method used was back-reading of already 
scored responses. This allowed the scoring supervisor to look at responses by category. The scoring 
supervisor also could review responses either by scorer or by score point agreements or splits. Scoring 
supervisors checked completion statistics as well. 


Cross-Country Scoring Reliability Study 

In international assessments, it is also important to gather information about how reliably the scoring was 
conducted from country to country so that valid international comparisons can be made of students’ 
achievement. To document the reliability of constructed-response scoring across both TIMSS and PIRLS 
countries, a cross-country scoring reliability study was conducted. Responses to TIMSS and PIRLS items 
from Southern Hemisphere countries were sent to the Northern Hemisphere countries for scoring. After 
the scoring of the U.S. responses for an item with their team, two scorers then completed scoring of the 
international responses that were preloaded on desktop computers. After training and successfully scoring 
a portion of the 2011 responses, select scorers were asked to score all the responses for an item. While the 
individuals making up the pairs varied, no scorer worked on items that he or she had not been trained to 
score. 


Trend Scoring 

To document the reliability of constructed-response scoring from TIMSS 2007 to TIMSS 2011 and 
PIRLS 2006 to PIRLS 2011, both TIMSS and PIRLS included a trend scoring reliability study. Lor each 
assessment, the study estimates the degree of agreement between the 2011 scorers and the scorers from 
the previous assessment. It allows scorers of the TIMSS and PIRLS 2011 assessment to score student 
responses collected in 2007 and 2006, respectively, and compare their scores to those given in TIMSS 
2007 and PIRLS 2006. The TIMSS and PIRLS 2011 scorers scored a subset of student responses from the 
TIMSS 2007 or PIRLS 2006 instruments. Student responses were actual student responses to items 
collected during the TIMSS 2007 and PIRLS 2006 assessments in the U.S. 
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The lEA Data Processing Center (DPC) assembled a sample of student responses to be scored and 
distributed it to each participating education system along with the lEA Trend Scoring Reliability 
Software (TSRS). Only education systems participating in 201 1 that also participated in TIMSS 2007 or 
PIRES 2006 took part in the trend scoring reliability study. There were 44 grade 4 items and 50 grade 8 
items for TIMSS trend scoring. There were 33 PIRES items for trend scoring. 

A qualification set of 50 responses for each selected trend item for both TIMSS and PIRES (or item part) 
was assigned to each scorer. After 25 responses were scored per item (item part), the trend reliability 
between the trend item was evaluated. The goal was to have between 85 percent to 100 percent agreement 
among scorers. If agreement for any item was below 85 percent for a particular scorer, retraining was 
required for the corresponding scorer and item. The scorer was then required to score the entire 
qualification set of 50 responses, rescoring the initial set of 25. If the scorer’s agreement level reached 85 
percent then the scorer began scoring 2011 student responses and the trend reliability set of responses. 

A total number of responses (200) were divided equally by the number of scorers who were assigned to 
the item block. The main trend reliability scoring process took place in parallel with the main study 
scoring for TIMSS and PIRES 2011. 


Scanning the Student Questionnaire Responses 

Responses to the TIMSS and PIRES student questionnaire items were scanned with optical-scanning 
equipment that also captured images of the constructed-response items and intelligent character 
recognition (ICR) fields. The data values captured from booklets and questionnaires were coded as 
numeric data. Unmarked fields were coded as blanks and the editing staff was alerted to missing or 
encoded critical data. The images of constructed-response items were saved as a digitized computer file. 
In addition to capturing the student responses, the barcode identification numbers used to maintain 
process control were decoded and transcribed to the TIMSS and PIRES computerized data file. ICR was 
used to read various hand and machine printing on the assessment booklets and student questionnaires. 
Image clips of the fields on the booklet covers were displayed to online editing staff for verification. 
Image clips were also taken for the open-ended items to be used for scoring. 
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Key-Entry of Teacher and School Questionnaire Responses 


For school and teacher questionnaires received as hard copy, the questionnaire data were entered directly 
into the online questionnaire for that school or teacher. 


Data Validation 

student booklets. Each data set produced by the scanning system was validated for type and range of 
response. The data-entry and resolution system used was able to simultaneously process a variety of 
materials from all age groups, subject areas, and assessment booklets as the materials were submitted to 
the system from scannable media. 

The data records in the scan file were organized in the same order in which the paper materials were 
processed by the scanner. As the program processed each record within a batch from the scan file, it wrote 
the edited and reformatted data records to the pre-edit file and recorded all errors on the edit file. The 
program generated an online edit file of the data problems and resolution guidelines. Image clips 
requiring edits were routed to online editing stations for the imaged scanned documents. 

All data values that were out of range were read “as is” but were flagged as suspect. All data fields that 
were read as asterisks (*) were recorded on the online edit file. Since the asterisk code indicated a double- 
response, these items were identified for possible resolution by editing staff. Each field was validated for 
range response and any values outside of the specified range. Corrections were made immediately. The 
system employed an edit/verify system, which meant two different people viewed the same suspect data 
and operated on it separately. The verifier made sure the two responses (one from either the entry operator 
or the ICR engine) were the same before the system accepted the item as being correct. If it could not be 
determined, it was escalated to a supervisor. 

When the edit process produced an error-free file, the booklet ID number was posted to the TIMSS and 
PIRLS tracking file by grade and school. This permitted staff to monitor the TIMSS and PIRES 
processing effort by accurately measuring the number of documents processed. The posting of booklet 
IDs also ensured that a booklet ID was not processed more than once. 
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File Creation and Consistency Checks 


In a final step the data from the assessment score files were merged with the student scanned data. At this 
time, final output files were produced for each file type. The final files were checked to ensure the data 
were in the correct format. In earlier editing functions, data were checked for completeness and 
compliance with codebook specifications. In addition, a check was performed to verify correct linking 
and matching of student, teacher, and school data files. Student and teacher files were loaded in the 
WinDem software such that all data from the assessments and questionnaires were available in the format 
required by lEA. 


4.6 Data Preparation 

As noted in the previous section, the data collected for TIMSS and PIRLS 2011 were entered into data 
files according to a co mm on international format, as specified in the WinDEM data entry software. The 
software facilitated the checking and correction of data by providing various data consistency checks 

The data files in this format were sent to the lEA Data Processing Center (DPC), where they were 
subjected to an extensive series of data cleaning and consistency checks. The overriding concern of these 
checks was to ensure that all information in the database conformed to the internationally defined data 
structure, national adaptations to questionnaires were reflected appropriately in the codebooks and 
documentation, and all variables used for international comparisons were comparable across countries. 


International Data Cleaning Procedures 

The DPC was responsible for checking the data files from each country, applying standard cleaning rules 
to verify the accuracy and consistency of the data, and documenting electronically any deviations from 
the international file structure. Queries arising during this process were addressed to national centers, and 
this process was repeated as necessary to ensure the data were consistent and comparable within and 
between countries. 

Following this cleaning step, countries were provided national univariate and reliability statistics along 
with data almanacs containing international univariate statistics and item statistics. This allowed countries 
to examine their data with those of other participating nations. Once any problems arising from this 
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examination were resolved, sampling weights produced by Statistics Canada and IRT-scaled student 
proficiency scores in mathematics and science were added to the fde. 

Detailed information on the entire data entry and cleaning process can be found in Martin and Mullis 

( 2011 ). 

Data Confidentiality Safeguards 

While the National Center for Education Statistics (NCES) and data contractors routinely pledge 
confidentiality to respondents, over the past decade concerns about the potential for disclosure of 
information about individual survey respondents have increased. These concerns are reflected in new laws 
enacted since the Privacy Act of 1974 to further ensure the protection of confidential data. The Education 
Sciences Reform Act of 2002 explicitly requires that NCES protect the confidentiality of all those 
responding to NCES-sponsored surveys so that no individual respondent can be identified. More 
specifically, NCES Standard 4-2, Maintaining Confidentiality (NCES, 2002), provides guidelines for 
limiting the risk of data disclosure for data released by NCES. Data disclosure occurs when an individual 
respondent has been identified through the use of the survey item responses and other external data 
sources. The procedures used to reduce the risk of data disclosure for TIMSS and PIRES 2011 in 
accordance with the guidelines specified in NCES Standard 4-2, are described below. 

All students, teachers, and schools participating in TIMSS and PIRES 2011 do so with the assurance that 
their identities will not be disclosed. Confidentiality procedures in place included the following: 

■ All employees with access to the data signed affidavits of data confidentiality. 

■ Questionnaires were sealed by students after completion. 

■ Names of students, teachers and schools were removed by field staff from the 
assessment booklets, the questionnaires, and all other related materials, and replaced 
with unique identification numbers. 

In addition to data collected directly from schools, teachers, and students, additional information was used 
during the TIMSS and PIRES sampling, data collection, and weighting processes, and these variables also 
were considered as part of the review to determine disclosure risk levels. 
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The confidentiality analysis review described below used the following three-step process to reduce 
disclosure risk: 

■ Determining the disclosure risk arising from existing external data; 

■ Creating a derived variable for race; and 

■ Swapping the data. 

In this process additional assurance is provided that individual schools, teachers, and students 
participating in TIMSS or PIRLS could be not identified through comparison with public data collections 
once the TIMSS data are released for public use. While no public data collections identify students or 
teachers by name, three publicly available data collections do identify schools by name. These are the 
Common Core of Data (CCD), a detailed public school listing; the Private School Survey (PSS), a 
detailed private school listing; and the Quality Education Data (QED) produced by Market Data Retrieval 
(MDR), a privately owned education research firm. The QED data contains a school-based file that 
provides demographic information for both public and private schools along with the names of the 
schools. Thus, there is some possibility that schools at least, and perhaps teachers and students as well, 
could be identified if comparisons of these data sets with the TIMSS data set allowed the identification of 
schools. 

Eeaving aside the question of why anyone would want to do this, it might be possible to identify TIMSS 
and PIRLS schools by taking variables from the TIMSS and PIRLS school data and searching the 
publicly available data files (QED, CCD, and/or PSS files) for schools with a matching profile. However, 
because the variables in the TIMSS and PIRLS data files were obtained from responses to the school 
questionnaire, for the most part, exact profile matches are unlikely. Even then, one would not know for 
certain whether any of the matched schools were the actual TIMSS and PIRLS schools or whether the 
match had simply arisen by chance. 

Nevertheless, school matching analyses were undertaken using probabilistic matching algorithms 
approved by the Institute of Education Sciences Disclosure Review Board (DRB) for use in disclosure 
analyses. These algorithms identify schools with some potential for identification. To provide further 
protection, elements of the data from schools identified as “disclosure risks” in this way were perturbed 
using the procedures approved by the DRB. After perturbation, the data were subjected to another round 
of analyses to ensure that the potential for identification no longer existed. 
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An additional measure was taken to reduce further the risk of disclosure of an individual respondent. This 
measure is referred to as data swapping, a DRB requirement that reduces risk by modifying microdata. In 
data swapping, a probability sample of records is paired with other records on the fde using selected 
characteristics, and then some identifying variables are swapped between pairs of records (see Kaufman 
et ah, 2005). The sampling rate for TIMSS and PIRLS swapping was designed to protect the 
confidentiality of the data without affecting the usability of the data set. All questionnaire data (school, 
teacher, and student) were involved in the swapping. This method is an effective way of keeping as much 
valuable data as possible while protecting respondent identity. Swapping preserves the univariate 
frequencies, means, and variances, although it may affect multivariate relationships a little. Pre- and post- 
swapping percentage distributions (unweighted and weighted) and correlations were reviewed to ensure 
data quality was maintained. 

Confidentiality analyses of this kind were conducted before the U.S. data files were delivered to the DPC 
for cleaning, and prior to the IRT scaling and estimation of sampling weights. 


The Estimation of TIMSS and PIRLS Student Proficiencies 

All cycles of TIMSS and PIRLS used item response theory (IRT) methods to produce score scales that 
summarized the achievement results. With this method, the performance of a sample of students in a 
subject area or sub-area could be summarized on a single scale or a series of scales, even when different 
students had been administered different items. 

IRT scaling provides estimates of item parameters (for example, item difficulty and item discrimination) 
that define the relationship between the item and the underlying variable measured by the test. Parameters 
of the IRT model are estimated for each test item, with an overall scale being established as well as scales 
for each content area and cognitive domain specified in the assessment framework. 

To allow for the calculation of trends in achievement, comparisons of scores across the five TIMSS 
assessments conducted in 1995, 1999, 2003, 2007, and 2011 and comparisons of scores across the PIRLS 
assessments conducted in 2001, 2006, and 2011 were necessary. To this end achievement scores from all 
five TIMSS assessments were placed on the same scale. The three PIRLS cycles were subjected to the 
same treatment. Details are provided in Methods and Procedures in TIMSS and PIRLS 2011 (Martin & 
Mullis, 2011). 
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Plausible Values 


During the scaling phase, plausible values were used to characterize scale scores for students participating 
in the assessment. To keep student burden to a minimum while ensuring content coverage, TIMSS and 
PIRLS administered a limited number of assessment items to each student — too few to produce accurate 
scale scores for each student. To account for this, TIMSS and PIRLS generated five possible scale scores 
for each student, each representing a random selection from the distribution of scale scores of students 
with similar backgrounds who answered the assessment items the same way. 

This plausible-values methodology was used to represent what the true performance of an individual 
might have been, had it been observed. This is done by using a small number of random draws from an 
empirically derived distribution of score values based on (a) the student’s observed responses to 
assessment items and (b) background variables. Each random draw from the distribution is considered a 
representative value from the distribution of potential scale scores for all students in the sample who have 
similar characteristics and identical patterns of item responses. The draws from the distribution are 
different from one another to quantify the degree of precision (the width of the spread) in the underlying 
distribution of possible scale scores that could have caused the observed performances. The TIMSS and 
PIRLS plausible values function like point estimates of scale scores for many purposes, but they are 
unlike true point estimates in several respects. They differ from one another for any particular student, 
and the amount of difference quantifies the spread in the underlying distribution of possible scale scores 
for that student. 

This approach to the estimation of scale scores ensures that the estimates of the average performance of 
student populations and the estimates of variability in those estimates are more accurate than those 
determined through traditional procedures, which estimate a single score for each student. An accessible 
treatment of the derivation and use of plausible values can be found in Beaton and Gonzalez (1995). A 
more technical treatment can be found in Methods and Procedures in TIMSS and PIRLS 2011 (Martin & 
Mullis, 2011). 


International Benehmarks 

International achievement benchmarks were developed to provide a concrete interpretation of what the 
scores on the TIMSS mathematics and science achievement scales and on the PIRLS reading scales mean. 
TIMSS and PIRLS each used scale anchoring to summarize and describe student achievement at four 
points on the mathematics, science, and reading scales — Advanced (625), High (550), Intermediate (475), 
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and Low (400). Scale anchoring involves selecting benchmarks (scale points) on the TIMSS and PIRLS 
achievement scales to be described in terms of student performance and then identifying items that 
students scoring at the anchor points can answer correctly. Subsequently, these items are grouped by 
content area within benchmarks and reviewed by mathematics, science, and reading experts. These 
experts focus on the content of each item and describe the kind of mathematics, science, or reading 
knowledge demonstrated by students answering the item correctly. The experts then provide a summary 
description of performance at each anchor point leading to a content-referenced interpretation of the 
achievement results. Detailed information on the creation of the benchmarks is provided in the “Using 
Scale Anchoring to Interpret the TIMSS and PIRLS 2011 Achievement Scales” section of Methods and 
Procedures in TIMSS and PIRLS 2011 (Martin & Mullis, 201 1). 


The Estimation of Sampling Weights 

Because of the complex sampling design used in TIMSS and PIRLS, students were assigned sampling 
weights. In general the sampling weight assigned to a student is the inverse of the probability that the 
student would be selected for the sample. When responses are weighted, each contributes to the results for 
the total number of students represented by the individual student assessed. Weighting also adjusts for 
school and student nonresponse. The internationally defined weighting specifications for TIMSS and 
PIRLS require that each assessed student’s sampling weight should be the product of 

■ the inverse of the school’s probability of selection; 

■ an adjustment for school-level nonresponse; 

■ the inverse of the classroom’s probability of selection; and 

■ an adjustment for student-level nonresponse. 

Sampling weights should be used in all TIMSS and PIRLS analyses. A detailed description of this process 
is provided in TIMSS 2011 User Guide for the International Database (Toy, Arora, & Stance, 2013) and 
the PIRLS 2011 User Guide for the International Database (Foy & Drucker, 2013). 
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5 . 


The TIMSS and PIRLS 2011 Data for the United 
States and Benchmarking States 


The TIMSS and PIRLS 2011 international databases contain student achievement data as well as student, 
teacher, school, and curricular background data for 54 countries and 1 1 other education systems for 
TIMSS, and 40 countries and 12 other education systems for PIRLS 2011. These databases provide 
comparable data across education systems on detailed measures of student achievement in mathematics 
and science for TIMSS participating education systems'® detailed measures of achievement in reading for 
PIRLS participants; information on educational practices and student outcomes; links between student 
achievement and background information from students, teachers, school principals, and curriculum 
experts; and achievement scales on a metric that is common to all cycles of TIMSS and PIRLS 
respectively, allowing for the analysis of trends. 


5.1 U.S. International, National, and State Benchmarking Data 
Files 


The TIMSS and PIRLS 2011 national data for the United States exist in the following three forms: 


■ U.S. international data files, which are part of the TIMSS and PIRLS international 
database and are directly comparable to that of other nations. As such these files allow 
for comparisons of the United States with any of the other education systems 
participating (except participating U.S. states) in TIMSS and/or PIRLS in virtually all 
respects. These files are available from the TIMSS and PIRLS International Study 
Center as SAS export files or SPSS “.sav” files through 
http://timssandpirls.bc.edu/timss201 1/intemational-database.html (TIMSS) or 
http://timssandpirls.bc.edu/pirls201 1/intemational-database.html (PIRLS). Note that 
these data files do not include the U.S. -specific adaptations made to a few questions in 
the questionnaires or the additional questions added to the school and student 
questionnaires, such as the question on race/ethnicity added to the student 
questionnaire. Data for U.S. states are not included in the international data files due 
to potential confidentiality issues. 

■ U.S. national public-use data files, which include the U.S. -specific adaptations that 
are not part of the U.S. international data files. These adaptations affect only a few 
variables and take the form of some elaboration of international response alternatives 
in a few items and the addition of questions to each of the questionnaires. These 
additional questions are described in section 5.9.2 of this report. The U.S. -specific 
TIMSS data files are available separately from the National Center for Education 
Statistics by download from http://nces.ed.gov/timss/datafiles.asp . The U.S. -specific 


"’Data for U.S. states are not included in the international data sets due to data confidentiality issues. 
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PIRLS data files are available separately from the National Center for Education 
Statistics by download from http ://nces . ed. go v/surveys/pirls/datafiles . asp . Because the 
U.S. national files differ little from their international counterparts, the bulk of the 
variables describing U.S. students, teachers, and schools are as described in the TIMSS 
2011 User Guide for the International Database (Foy, Arora, & Stanco, 2013) and the 
PIRLS 2011 User Guide for the International Database (Foy & Drucker, 2013). These 
are the most comprehensive and detailed references for the TIMSS and PIRFS 2011 
data and should be seen as the primary reference. 

■ U.S. national restricted-use data files, which can only be obtained by completing a 
restricted-use license agreement with NCES. The restricted-use data files are provided 
only on CD. These datasets contain the supplemental link files that link TIMSS or 
PIRFS school ID numbers to the school ID numbers as they appear in the publicly 
available Common Core of Data (CCD) or the Private School Universe Survey (PSS). 
In addition, race/ethnicity is provided with all available categories and free or reduced 
price lunch is provided as a continuous variable. Because these data can reveal the 
identities of participating schools, the restricted-use data files are only made available 
to those who obtain a NCES restricted-use data license. Directions on how to obtain 
the license can be found at http://nces.ed.gov/pubsearch/licenses.asp . 


Data for the nine states that participated in TIMSS and/or PIRFS are provided in one format: 


■ State restricted-use data files, which can only be obtained by completing a 

restricted-use license agreement with NCES. The restricted-use data files are provided 
only on CD. Fike the U.S. national restricted-use data files, these datasets contain the 
supplemental link files that link TIMSS or PIRFS school ID numbers to the school ID 
numbers as they appear in the publicly available Common Core of Data (CCD) or the 
Private School Universe Survey (PSS). In addition, the state data files also contain the 
state -level data parallel to the U.S. national public-use files. Because these data can 
reveal the identities of participating schools, the restricted-use data files are only made 
available to those who obtain a NCES restricted-use data license. Directions on how 
to obtain the license can be found at http://nces.ed.gov/pubsearch/licenses.asp . 


The discussion that follows is designed to provide the following: 


■ a summary overview of sections of the TIMSS 2011 User Guide for the International 
Database and the PIRFS 2011 User Guide for the International Database as these 
relate to using the U.S. data files; 

■ documentation of U.S. national items that differ from their international versions in 
the U.S. and state files; and 

■ a description of U.S. national items that are unique to the U.S. national and state files. 
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5.2 


TIMSS and PIRLS Data Files 


The following five basic types of data files are available for each education system in both the TIMSS and 
PIRLS international datasets: 


■ achievement files containing item response data and scale scores for the TIMSS or 
PIRLS assessment; 

■ background files with information from students, from their mathematics and science 
teachers in TIMSS or their reading teachers in PIRLS, and from the principals of their 
schools; 

■ student-teacher linkage files that contain the information needed to link data on 
students to that of their teachers; 

■ constructed-response scoring reliability files providing data on the reliability of 
scoring for this type of item; and 

■ curriculum data files that contain the responses of countries or participating education 
systems to the curriculum questionnaires. 


The following discussion focuses on the first three categories of files as these are the ones most likely to 
be used in data analyses by most users. For the remaining two categories of files, the reader is referred to 
the TIMSS 2011 User Guide for the International Database (Foy, Arora, & Stanco, 2013) or the PIRLS 
2011 User Guide for the International Database (Foy & Drucker, 2013). 


Data File Naming Convention for TIMSS/PIRLS 

The file names of the data files consist of an eight-character string followed by a three-character file 
extension using the conventions listed below. These conventions are illustrated by reference to the SPSS 
file containing the fourth-grade school background data for the United States (ACGUSAMS.sav) and 
govern both TIMSS and PIRLS data file nomenclature. 

The first character of the file name indicates grade level (ACGUSAM5). 

■ A - fourth grade 

■ B - eighth grade 
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The second character indicates the source or level of the information in the file (ACGUSAM5). 


■ C - school-level fde 

■ T - teacher-level file 

■ S - student-level fde 

The third character indicates the type of data in the fde (ACGUSAM5). 

■ A - student achievement items 

■ G - general background questionnaire items (school and student questionnaires), and 
the general teacher questionnaire for the fourth grade 

■ M - mathematics teacher background questionnaire 

■ S - science teacher background questionnaire 

■ T - student-teacher linkage fdes 

Characters four through six identify the education system using a three-character alphanumeric education 
system code based on the ISO coding scheme (ACG USA M5). See exhibit 4-1 in Foy, Arora and Stanco 
(2013) for a list of the ISO codes for each participating education system in TIMSS and, for PIRLS, 
exhibit 4-1 in Foy and Drucker (2013). 

The seventh and eighth characters indicate the study cycle (ACGUSA M5) . 

■ Ml -TIMSS 1995 fdes 

■ M2 -TIMSS 1999 fdes 

■ M3 - TIMSS 2003 fdes 

■ M4 - TIMSS 2007 fdes 

■ M5- TIMSS 2011 fdes 

■ R1 -PIRLS 2001 fdes 

■ R2 - PIRLS 2006 fdes 

■ R3- PIRLS 2011 fdes 

'’Note that for PIRLS “A” “G” and “T” are the only characters used, as it is a single-subject assessment. To identify PIRLS files, the seventh and 
eighth characters must be considered. 
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The three-character file extensions used for the data files are as follows: 


■ .EXP - SAS export data 

■ .SAV - SPSS data 


Exhibit 5-1 and 5-2 show the formats of the TIMSS 2011 and PIRLS 2011 data fde names according to 
this scheme. For example, ASGUSAM5.SAV is an SPSS file that contains the U.S. TIMSS 2011 fourth- 
grade student background data, and BTSUSAM5.EXP is an SAS export file that contains the U.S. TIMSS 
2011 eighth-grade science teacher background data. For each file type, a separate data file is provided for 
each participating education system. Data files are distinguished by the ISO code in positions four 
through six. 

Exhibit 5-1. TIMSS data file names 


File Names 

Descriptions 

ACG***M5 

ASA*#*M5 

ASG***M5 

AST***M5 

ATG***M5 

BCG***M5 

BSA**#M5 

BSG***M5 

BST##*M5 

BTM*#*M5 

BTS#**M5 

Fourth-grade school background data files 
Fourth-grade student achievement data files 
Fourth-grade student background data files 
Fourth-grade student-teacher linkage files 
Fourth-grade teacher background data files 
Eighth-grade school background data files 
Eighth-grade student achievement data files 
Eighth-grade student background data files 
Eighth-grade student-teacher linkage files 
Eighth-grade mathematics teacher background data files 
Eighth-grade science teacher background data files 

• •• indicates three-character alphanumeric education system code (based on the ISO coding scheme) 
that is part of filename. 

SOURCE: TIMSS 2011 User Guide for the International Database, exhibit 4-2, p. 81. 

Exhibit 5-2. PIRLS data file names 

File Names 

Descriptions 

ACG***R3 

ASA**#R3 

ASG***R3 

AST***R3 

ATG*#*R3 

Fourth-grade school background data files 
Fourth-grade student achievement data files 
Fourth-grade student background data files 
Fourth-grade student-teacher linkage files 
Fourth-grade teacher background data files 


• •• indicates three-character alphanumeric education system code (based on the ISO coding scheme) 
that is as part of filename. 

SOURCE: PIRLS 2011 User Guide for the International Database, exhibit 4-2, p. 81. 
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5.3 


TIMSS and PIRLS Achievement Data Files and Variable 
Names 


The data files containing the IRT-scaled achievement scores for overall mathematics and science, the 
several mathematics and science content domains, and the three mathematics and science cognitive 
domains are identified by the first three characters in the file name. A set of five plausible values 
characterizes each of these achievement scores. 

Files beginning with ASA are fourth-grade achievement score files, and those beginning with BSA are 
eighth-grade achievement score files. For analytic convenience, these same achievement scores are also 
provided as an addition to the student background data files. 

The achievement score variable names are based on an eight-character string defined below. In exhibit 5- 
3 these conventions are illustrated by reference to the first plausible value for each of the total, content 
domain, and cognitive domain achievement scales at each grade level. 

First character of the variable name: 

■ A - fourth-grade score 

■ B - eighth-grade score 

Second character of the variable name: 

■ S - indicates that this is a student score variable 
Third character of the variable name: 

■ M - mathematics score 

■ S - science score 

■ R - reading score 

Fourth through sixth characters of the variable name: 

■ Three-character code identifying the achievement scale, as indicated in exhibit 5-3 
below. 
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Seventh and eighth characters of the variable name: 


■ 0 1 - first plausible value 

■ 02 - second plausible value 

■ 03 - third plausible value 

■ 04 - fourth plausible value 

■ 05 - fifth plausible value 

For example, ASMMATOl is the first plausible value for the fourth-grade mathematics total score. Again, 
ASSSC102 is the second plausible value for the total science score, ASSEAR03 is the third plausible 
value for the content domain “Earth science,” and ASREIT05 is the fifth plausible value for domain 
“Reading for Literacy Experience 

Exhibit 5-3 shows the nomenclature for identifying the various total, content domain, and cognitive 
domain achievement scale score variables, each of which is represented by five plausible values. 


TIMSS and PIRLS Benchmark Achievement Variables 

The TIMSS and PIRLS achievement files also contain a set of variables indicating which international 
benchmark the students reached. For TIMSS, the overall mathematics and science scales at both grades 
have five plausible values for each of the four benchmark levels defined ( advanced, high, intermediate, 
low). For PIRLS, a similar set of plausible values is provided for the overall reading scale. The 
international benchmark variables follow the achievement score variable naming convention but 
substitute the letters “IBM” in the fourth through sixth positions of the variable name. Thus, ASMIBMOI, 
ASM1BM02, ASM1BM03, ASM1BM04, and ASM1BM05 are the five benchmark variables describing 
the fourth-grade overall mathematics score. Similarly, BSMIBMOI through BSM1BM05 describe the 
eighth-grade overall mathematics score and BSSIBMOI through BSS1BM05 describe the eighth-grade 
overall science score. Details for TIMSS are provided in Foy, Arora and Stanco (2013, pp. 86-87) and for 
PIRLS in Foy and Drucker (2013, pp. 86-87). 
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Exhibit 5-3. Three-digit codes identifying achievement scales in TIMSS and PIRLS 



Score 

Identifier 

Examples of variable 
Grade 4 

names 

Grade 8 

Mathematics overall score 

MAT 

ASMMATOl 

BSMMATOl 

Science overall score 

SCI 

ASSSClOl 

BSSSClOl 

Reading overall score 

REA 

ASRREAOl 


Mathematics content domains 

Number 

NUM 

ASMNUMOl 

BSMNUMOl 

Geometric Shapes/Geometry 

GEO 

ASMGEOOl 

BSMGEOOl 

Data Display/Data and Chance 

DAT 

ASMDATOl 

BSMDATOl 

Algebra 

AEG 


BSMALGOl 

Science content domains 

Life Science 

LIE 

ASSLIFOl 


Physical Science/Physics 

PHY 

ASSPHYOl 

BSSPHYOl 

Earth Science 

EAR 

ASSEAROl 

BSSEAROl 

Chemistry 

CHE 


BSSCHEOl 

Biology 

BIO 


BSSBIOOl 

Mathematics cognitive domains 

Knowing 

KNO 

ASMKNOOl 

BSMKNOOl 

Applying 

APP 

ASMAPPOl 

BSMAPPOl 

Reasoning 

REA 

ASMREAOl 

BSMREAOl 

Science cognitive domains 

Knowing 

KNO 

ASSKNOOl 

BSSKNOOl 

Applying 

APP 

ASSAPPOl 

BSSAPPOl 

Reasoning 

REA 

ASSREAOl 

BSSREAOl 

Purposes for Reading 

Reading for Literary Experience 

LIT 

ASRLITOI 


Reading to Acquire and Use Information 

INF 

ASRINFOl 


Processes of Comprehension 
Retrieving and Straightforward 

Inferencing 

RSI 

ASRRSlOl 


Interpreting, Integrating, and Evaluating 

HE 

ASRllEOl 



SOURCE: Foy, P„ Arora, A., and Stance, G. (Eds.) (2013). TIMSS 2011 User Guide for the 
International Database. Chestnut Hill, MA: Boston College; Foy, P. and Drucker, K.T. (Eds.) (2013). 
PIRLS 2011 User Guide for the International Database. Chestnut Hill, MA: TIMSS & PIRLS 
International Study Center, Boston College 
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5.4 


TIMSS and PIRLS Background Questionnaire Data Fiies 


Student, teacher, and school fdes contain the responses to the questions contained in the respective 
background questionnaires administered in TIMSS 2011 or PIRLS 2011, along with a fourth fde used to 
link the student and teacher background data appropriately when student and teacher fdes are merged. 


TIMSS and PIRLS Student Background Data Files (ASG/BSG) 

The student background data fdes contain students’ responses to questions in the student questionnaire 
along with students’ mathematics and science achievement scores (as plausible values). At the fourth 
grade, there was a single version of the student questionnaire for TIMSS and PIRLS. However, 
internationally there were two versions of the student questionnaire at the eighth grade: a version for 
education systems in which science is taught as an integrated subject (general science version), and 
another version for education systems in which the sciences (biology or life science, physics, chemistry, 
and Earth science) are taught separately. For eighth-grade students who were administered the general 
science version, as was the case for the United States and benchmarking U.S. states, questions that 
appeared only in the separate science version were coded as “not administered.” For students in those 
education systems assigned the separate science versions, questions asked only in the general science 
version were coded as “not administered.” 

The student background data files also contain a number of identification variables, tracking variables, 
sampling and weighting variables, and derived variables that were used to produce some of the exhibits in 
the international reports. 


TIMSS and PIRLS Teacher Background Data Files (ATG/BTM/BTS) 

The mathematics and science teachers of the students sampled for TIMSS 2011 were administered at least 
one questionnaire for each TIMSS class taught. The teacher background data files contain one record for 
each of the classes taught. If teachers taught more than one class, they were expected to complete only 
one set of general background questions (part A), irrespective of the number of classes taught, and a 
separate part B (class-specific questions) for each class they taught. 

Teachers of PIRLS classes were administered a single teacher background questionnaire that had 
questions about their background and their teaching practices in the classes of sampled students. Each 
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teacher was asked to complete a questionnaire for each class taught that contained sampled students. The 
PIRLS teacher background data files contain one record for each teacher and class combination. 

Separate TIMSS teacher questionnaires were administered to eighth-grade mathematics and science 
teachers. The responses of teachers to the mathematics questionnaire are found in the BTM files, and the 
responses of teachers to the science questionnaire are in the BTS files. Variable names for questions 
repeated in both questionnaires are the same. Responses to the single questionnaire administered to 
fourth-grade teachers of TIMSS and PIRLS are found in the ATG files. The seventh and eighth characters 
(M5 or R3) discern between TIMSS and PIRLS teacher data files. 

In all of the teacher fdes each teacher has a unique identification number (IDTEACH) and a link number 
(IDLINK) specific to the class taught by the teacher and to which the information in the data record 
corresponds. The IDTEACH and IDEINK combination uniquely identifies, within an education system, a 
teacher teaching a specific class. For example, students linked to teachers identified by the same 
IDTEACH but different IDEINK are taught by the same teacher but in different classes. It is important to 
note that the teachers in question do not constitute a representative sample of teachers in an education 
system but rather are the teachers who taught a representative sample of students. To reflect this fact, for 
the most part, the teacher data should be analyzed only in conjunction with the student-teacher linkage 
data files and weighted with student sampling weights. 


TIMSS and PIRLS School Background Data Files (ACG/BCG) 

The school background data files contain the responses of school principals to questions about school 
policy, resources, and environment asked in the school questionnaire. That file also contains a series of 
identification variables, link variables, and sampling variables. The school data files can be merged with 
the student data files by using the education system and school identification variables. Details of the 
merging procedure using the SPSS-linked lEA International Database (IDB) Analyzer or using SAS 
programs for TIMSS are described in the TIMSS 2011 User Guide for the International Database (Foy, 
Arora, & Stanco, 2013) and for PIRLS these details are provided in the PIRLS 2011 User Guide for the 
International Database (Foy & Drucker, 2013/ 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


139 


TIMSS and PIRLS Student-Teacher Linkage Data Files (AST/BST) 


The TIMSS and PIRLS 2011 student-teacher linkage data files contain information required to link the 
student and teacher data files. These files contain one entry per student-teacher linkage combination in the 
data. For instance, if three teachers are linked to a student, there are three entries in the file corresponding 
to that student. The sole purpose of the student-teacher linkage data files is to link teacher-level data with 
student-level data to perform appropriate student-level analyses where teacher characteristics are 
disaggregated over students. 


TIMSS and PIRLS Curriculum Questionnaire Data Files 

In addition to the background questionnaires, TIMSS and PIRLS also provides data on the curriculum of 
the participating education systems. For TIMSS there is a separate file for each grade, while for PIRLS 
there is a single file. 


5.5 Variable Naming Convention for Background Variables 

The background variable naming convention is based on a seven- or eight-character string defined below. 
These conventions are illustrated by reference to an item in the fourth-grade school questionnaire. This 
item asks principals to report the population size of the community in which the school is located. 

First character of the variable name (ACBG05A): 

■ A - fourth-grade 

■ B - eighth-grade 

Second character of the variable name (ACBG05A): 

■ C - school principal 

■ T - teacher 

■ S - student 
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Third character of the variable name (ACBG05A): 


■ N - a national or nationally adapted background variable 

■ B - all background variables in the questionnaire data fdes 

■ D - all derived variables 

Fourth character of the variable name ^^ (ACBG05A): 

■ G - general question (not subject specific)* 

■ R - question related to reading 

■ M - question related to mathematics 

■ S - question related to science 

■ B - question related to biology or life science 

■ C - question related to chemistry 

■ E - question related to earth science 

■ P - question related to physics of physical science 

Fifth through eighth characters of the variable name (ACBG 05A) : 

■ Used to represent the sequential numbering of each question 

For PIRLS, the curriculum questionnaire uses a different variable naming convention. The first three 
characters are “GEN” for general questions, or “REA” for questions relating to reading instruction. The 
remaining characters are defined by the sequential location of the questions in the questionnaire (Foy & 
Drucker, 2013). 


Summary Indices and Derived Variables 

The TIMSS and PIRLS questionnaires often devote several questions to a single construct. In these cases, 
responses to the individual items were combined to create a derived variable. A TIMSS or PIRLS index is 
a special type of derived variable that assigns students to one of three levels — high, medium, or low — on 


'*The letters “B,” “C,” “E,” and “P” are used only in the eighth-grade student backgroxmd data files for variables corresponding to questions 
about separate sciences asked in the separate science version of the student questionnaire. 
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the basis of their responses to the component variables. These variables are described in detail in 
Supplement 3 of the PIRLS 2011 User Guide for the International Database (Foy & Drucker, 2013) and 
in Supplement 3 of the TIMSS 2011 User Guide for the International Database (Foy, Arora, & Stanco, 
2013). 

5.6 Sampling and Weighting Variables 

Several sampling and weighting variables are included in the TIMSS and PIRLS data files. They are 
listed and described below in conjunction with a discussion of how and when these weights are used. 
Because TIMSS and PIRLS use a complex sampling design, sampling weights need to be used to 
generate accurate population estimates. The sampling weights account for the sample design, any 
stratification or disproportional sampling of subgroups, and also include adjustments for nonresponse (see 
Joncas & Foy, 2012). 

As noted, the sample of students is not a simple random sample and, as a consequence, students in the 
sample do not have an equal probability of selection. Sampling weights adjust for this unequal probability 
and, in so doing, provide for statistical estimates reflective of the student population from which the 
sample was drawn. Sampling weights also include adjustments for school and student nonresponse. All 
TIMSS and PIRLS analyses require the application of sampling weights. Provisions for weighting data 
are a standard feature of virtually all software likely to be used in analyses. 

The sampling weights included in the TIMSS and PIRLS 2011 data files are described in exhibit 5-4. 
(Note that teacher background data files do not have any sampling weight variables since the analysis of 
teacher variable requires the merging of the teacher data with the student data and the use of student 
sampling weight variables.) 
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Exhibit 5-4. TIMSS and PIRLS sampling weight variables 


Variable Names 

Descriptions 

TOTWGT 

Total student weight - sums to the national population 

SENWGT 

Student senate weight - sums to 500 in each education system 

HOUWGT 

Student house weight - sums to the student sample size in each education system 

TCHWGT 

Overall teacher weight 

MATWGT 

Mathematics teacher weight 

SCIWGT 

Science teacher weight 

SCHWGT 

School-level weight 


SOURCE: Foy, P„ Arora, A., and Stance, G. (Eds.) (2013). TIMSS 2011 User Guide for the 
International Database. Chestnut Hill, MA: Boston College; Foy, P. and Drucker, K.T. (Eds.) (2013). 
PIRLS 2011 User Guide for the International Database. Chestnut Hill, MA: TIMSS & PIRLS 
International Study Center, Boston College 


The characteristics of TIMSS and PIRLS sampling weight variables are as follows: 


■ TOTWGT sums to the student population size in each education system and is 
appropriate for “within-country”'^ analyses and “cross-country” analyses where the 
analyses are conducted “country-by-country” and compared. 

■ SENWGT is a transformation of TOTWGT that results in a weighted student sample 
size of 500 in each education system. This weight may be appropriate for cross- 
"country” analyses that require each education system to have the same number of 
students rather than proportionately more students from larger education systems and 
fewer from smaller education systems, which is the case if TOTWGT is used. 

■ HOUWGT, another transformation of TOTWGT, ensures that the weighted sample 
corresponds to the actual sample size in each education system. This can be important 
since TOTWGT inflates sample sizes to approximate the population size, and 
software systems that use the actual sample size to compute significance tests will 
give misleading results under these conditions. 

TOTWGT, SENWGT, and HOUWGT are designed for use in student-level 
analyses from all student-level files. 

SCHWGT is designed for use in school-level analyses where the schools are 
the units of analysis. 

TCHWGT, MATWGT, and SCIWGT are specifically designed for analyses 
that link teacher background data to student data. 


'^Although traditional terms with “country” are used throughout this discussion, they apply as well to all participating states and other education 
systems. 
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TCHWGT is used for analyses using all teachers. 

MATWGT and SCIWGT are used for analyses of mathematics and science 
teachers, respectively. 


5.7 Structure and Design Variables in TIMSS and PIRLS 2011 
Data Files 


The TIMSS and PIRLS 2011 data files also contain unique numerical identification variables for each 
respondent along with sample design information. 


Identification Variables 


In all TIMSS and PIRLS data files, identification variables are included to label countries, students, 
teachers, or schools. These variables also are used to link cases between the different data fde types. The 
identification variables are the same across PIRLS and TIMSS, have the prefix “ID,” and are described 
below. 


■ IDCNTRY : a five-digit country identification code based on the ISO 3166 
classification. 

■ IDPOP: identifies the target grade; “1” for the fourth grade and “2” for the eighth 
grade. 

■ IDGRADE: identifies the target grade of the participating students; “4” and “8” for 
most countries. 

■ IDSCHOOL: a four-digit identification code that uniquely identifies the participating 
schools within each country but are not unique across countries. 

■ IDCLASS: a six-digit identification code that uniquely identifies the sampled 
classrooms within a country. 

■ IDSTUD: an eight-digit identification code that uniquely identifies each sampled 
student in a country. 

■ IDBOOK: identifies the specific assessment booklet that was administered to each 
student. 

■ IDSTRATE & IDSTRATl: identification variables generated by the school sampling 
process. IDSTRATE identifies the explicit strata and IDSTRATl the implicit strata 
from which the participating schools were sampled. 
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■ IDTEACH: a six-digit identification code that uniquely identifies a teacher within a 
school. 

■ IDLINK: uniquely identifies the class for which a teacher answered a questionnaire. 


Tracking Variables 

Information about students, teachers, and schools provided by the survey tracking forms described earlier 
is stored in the tracking variables. These variables have the prefix “IT.” All tracking variables are 
included in the student background data files. ITLANG is included in the student achievement and student 
background data files. 

■ ITSEX: Gender of each student as stated in the Student Tracking Form. 

■ ITBIRTHM and ITBIRTHY: month and year of birth of each student as stated in the 
Student Tracking Forms. 

■ ITDATEM and ITDATEY : month and year of testing for each student. 

■ ITEANG: language of testing for each student. 


5.8 TIMSS and PIRLS 201 1 Codebook Files 

All information related to the structure of the TIMSS and FIRES 2011 data files, as well as the source, 
format, descriptive labels, and response option codes for all variables, is contained in codebook files. 
Each data file type in the database is accompanied by a codebook file, with the exception of the 
curriculum data files. These files are available from the TIMSS and FIRES International Study Center 
website at http://timssandpirls.bc.edu/timss201 1/intemational-database.html . 


5.9 The U.S. National and State Benchmarking Data Files 

The instrumentation for the U.S. national and state samples are identical because states were administered 
the U.S. national instruments. As noted earlier, the U.S. national instrumentation differs from the 
international instrumentation in five ways: 

■ Minor language/expression adaptations were made to some of the instructions. 

■ Minor language adaptations were made to the wording of some assessment items. 
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■ For a few questionnaire items, response alternatives were changed but in a way that 
allowed a crosswalk to the international response alternatives. 

■ One international question was not asked in the United States because of its 
sensitivity. 

■ Several U.S. -specific questions without international counterparts were added to the 
student and school questionnaires. 


Otherwise, the U.S. instrumentation is exactly the same as the international instrumentation. This will 
become apparent in comparisons between U.S. and international questionnaires. For standard 
international versions of TIMSS and PIRLS 2011 questionnaires, please see Supplement 1 of either 
TIMSS or PIRLS User Guide for the International Database. TIMSS and PIRLS provide parallel 
supplements to the International User Guides. 


For TIMSS, the supplements to the international user guide are as follows: 


■ Supplement 1, International Version of the TIMSS 2011 Background and Curriculum 
Questionnaire, can be downloaded from the TIMSS and PIRLS International Study 
Center at 

http://timssandpirls.bc.edu/timss201 1/downloads/Tl 1 UG Supplementl.pdf 

■ Supplement 2 for TIMSS can be downloaded from the TIMSS and PIRLS 
International Study Center at 

http://timssandpirls.bc.edu/timss201 1/downloads/Tl 1 UG Supplement2.pdf 

■ Supplement 3, the TIMSS Variables Derived from the Student, Home, Teacher, and 
School Questionnaire Data of the TIMSS 2011 User Guide for the International 
Database can be downloaded from the TIMSS and PIRLS International Study Center 
at http://timssandpirls.bc.edu/timss201 1/downloads/Tl 1 UG Supplement3.pdf 

■ The U.S. versions of the TIMSS questionnaires can be found at 
http://nces.ed.gov/timss/questionnaire.asp . The variable names in the U.S. files are 
identical to those in the international files with the exception of the U.S. -specific 
variables added. 


The PIRLS supplements to the PIRLS International User Guide are as follows: 


■ Supplement 1, International Version of the PIRLS 2011 Background Questionnaires 
and Curriculum Questionnaire, can be downloaded from the TIMSS and PIRLS 
International Study Center at 

http://timssandpirls.bc.edu/pirls201 1/downloads/Pl 1 UG Supplementl.pdf 

■ Supplement 2, like TIMSS, the PIRLS National Adaptations of International 
Background Questionnaires of the PIRLS 2011 User Guide for the International 
Database can be downloaded from the TIMSS and PIRLS International Study Center 
at http://timssandpirls.bc.edu/pirls201 1/downloads/Pl 1 UG Supplement2.pdf 
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■ Supplement 3, the PIRLS Variables Derived from the Student, Home, Teacher, and 
School Questionnaire Data of the PIRLS 2011 User Guide for the International 
Database can be downloaded from the TIMSS and PIRLS International Study Center 
at http://timssandpirls.bc.edu/pirls201 1/downloads/Pl 1 UG Sunplement3.pdf 

■ The PIRLS questionnaires may be downloaded from 
http://nces.ed.gov/survevs/pirls/questionnaire.asp . The variable names in the U.S. files 
are identical to those in the international files with the exception of the U.S. -specific 
variables added. 


Background Questionnaire Items With U.S. Adaptations to Response 
Alternatives 

As the description of U.S. national adaptations in appendix F makes clear, there were a number of 
relatively minor changes to the wording of the international item stems and response alternatives in the 
questionnaires. Most of these adaptations do not require comment, as they are identical in format between 
the international and U.S. versions of the questionnaires (for example, they contain simple wording 
changes). In some cases, however, the adaptations resulted in item response formats not immediately 
comparable between the international and national versions of the questionnaires. As indicated in 
appendix F, there are instances in which the international and U.S. versions of variables have different 
sets of response codes; “highest level of formal education” in the teacher questionnaire is one example. 
This means that, for these items, the data will not be identical in international and U.S. versions of the 
data files. Using the same example, “highest level of formal education” will have six response categories 
in the U.S. international file and seven categories in the U.S. national file.^° However, as indicated in 
appendix F crosswalks between international and U.S. versions of these questions allow for the 
conversion of the U.S. response codes to the international format. 


U.S. -Specific Variables 

U.S. -specific items were added to the student, teacher, and school questionnaires. Six questions were 
added to the student questionnaires: 

1 . a two-part question designed to measure the student’s race/ethnicity; 

2. a question that asked for language other than English spoken at home; 

’”The paper version of teacher questionnaire had 8 categories for this question, but was recoded to 7 categories to match the on-line version. This 
was recoded to 6 categories to match the international version of this question. 
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3. a 3 -part question asking about place of birth and parents place of birth; 

4. a question about additional activities outside of school; 

5. a question asking about number of days absent; and 

6. a two-part question that asked students to indicate whether they had repeated a grade 
in elementary and/or middle/junior high school (grade 8 student questionnaire only). 

One question was added to the grade 8 teacher questionnaires that asked teachers to identify the title of 
the mathematics or science course being taught to the students being assessed. Three questions were 
added to the school questionnaire: 

1 . the percentage of students in the school eligible for free or reduced-price lunch; 

2. the percentage of students in the school who are English language learners; and 

3. a specification of the type of school. 


Race/Ethnicity (TIMSS 4 & 8/PIRLS Student Questionnaire) 

Students’ race/ethnicity was obtained through student responses to a two-part question in the student 
questionnaire. Students were asked first whether they were Hispanic or Latino and then whether they 
were members of the following five racial groups: (1) American Indian or Alaska Native; (2) Asian; (3) 
Black or African American; (4) Native Hawaiian or other Pacific Islander; or (5) White. Multiple 
responses to the second of these questions were allowed.^' A composite variable with six categories was 
constructed in which results are shown separately for (1) Hispanics of any race; (2) Blacks; (3) Whites; 
(4) Asians; and (5) Multiracial. The sixth category was labeled as “Other” and consisted of the small 
numbers of students indicating that they were American Indian, Alaska Native, Native Hawaiian, or other 
Pacific Islander. 


^^Race-ethnicity is provided with all categories in the restricted-use data set . 

^^Race-ethnicity is provided as a composite variable in the public-use data set. The state benchmarking restricted-use datasets contain both the 
categorical and composite forms of the variable. 
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Language Other Than English Spoken at Home (TIMSS 4 & 8/PIRLS Student Questionnaire) 


This item extended the international question about how often students spoke English at home to ask 
those who indicated that they did not always speak English if they spoke Spanish or another language. 


Repeating a Grade (TIMSS 8 Student Questionnaire) 

Students were asked whether they had ever repeated a grade in either “elementary school” and/or “middle 
or junior high school.” The response alternatives were “yes” or “no” in each case. 


Additional Outside Aetivities (TIMSS 4/PIRLS Student Questionnaire) 

The measure of outside of school activities was collected using a prompt with 4 yes/no questions. The 
prompt states, “The following questions ask about the activities you do outside of school . The yes/no 
questions were as follows: 

■ Do you play on a sports team outside of school? 

■ Do you play a musical instrument outside of school? 

■ Are you studying something in a class outside of school? 

■ Do you belong to a club outside of school (like Girl Scouts, Cub scouts, 4-H, or Boys 
and Girls Club)? 


Number of Days Absent (TIMSS 4 & 8/PIRLS Student Questionnaire) 

TIMSS and PIRLS students were asked about their school attendance over the last month prior to the 
assessment. The students were asked to select one response option. The question and response options 
were as follows: 


grade 4 and 8, students who responded to items ASBG03N (grade 4) and BSBG03 (grade 8) that they “always” speak English at home were 
supposed to skip items ASNG03B (grade 4) and BSNG03B (grade 8), which asks students to identify the foreign language that they sometimes 
use at home. However, about 10 percent of students (1267 cases) at grade 4, and 5 percent of students (565 cases) at grade 8 responded that they 
“always” speak English at home and also identified a foreign language that they sometimes use at home. 
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How many days were you absent from school in the last month? 


■ None; 

■ 1 or 2 days; 

■ 3 or 4 days; 

■ 5 to 10 days; or 

■ More than 10 days. 


Country of Mother/Father/Student (TIMSS 4/PIRLS Student Questionnaire) 


Grade 4 students, in both TIMSS and PIRLS, were asked about the location of their birth and about the 
location of their parents or legal guardians birth using three yes/no questions. These questions were as 
follows: 


A. Was your mother (or stepmother or female legal guardian) bom in the United States? 
(“United States” includes the 50 states, its territories, the District of Columbia, and 
U.S. military bases abroad.) 

B. Was your father (or stepfather or male legal guardian) bom in the United States? 

C. Were you born in the United States? 


Mathematies Course Taught (TIMSS 8 Teaeher Questionnaire) 

Eighth-grade mathematics teachers were asked about the nature of the mathematics course they taught to 
the TIMSS students. The response alternatives provided were as follows: 

■ Basic or general eighth-grade math (not algebra or pre-algebra); 

■ Introduction to algebra or pre-algebra; 

■ Two-year pre-algebra; 

■ Algebra 1 ( 1 -year course); 

■ Algebra 1 (first year of a 2-year algebra 1 course); 

■ Algebra 1 (second year of a 2-year algebra 1 course); 
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■ Geometry; 

■ Algebra 11; 

■ Integrated or sequential math; and 

■ Other math class. 


Science Course Taught (TIMSS 8 Teacher Questionnaire) 

Eighth-grade science teachers were asked about the nature of the science course they taught to the TIMSS 
students. The response alternatives provided were as follows: 

■ General science (several content areas of science taught separately); 

■ Integrated science (several content areas of science taught combined and taught 
throughout the year); 

■ Life science (e.g., biology, ecosystems, human health); 

■ Physical science (e.g., physics or chemistry); and 

■ Earth science (e.g., geology, earth and the solar system, fossils). 


Poverty Levei in Pubiic Schoois (Percentage of Students Eiigibie for Free or Reduced-Price 
Lunch) (TIMSS 4 & 8/PIRLS Schooi Questionnaire) 

The measure of poverty level in public schools was obtained from principals’ responses to the school 
questionnaire. The question asked the principal to report, as of approximately October 20 1 0, the 
percentage of students at the school eligible to receive free or reduced-price lunch through the National 
School Lunch Program. Responses were grouped into five categories: (1) less than 10 percent, (2) 10 to 
24.9 percent, (3) 25 to 49.9 percent, (4) 50 to 74.9 percent, and (5) 75 percent or more.^"^ Missing data on 
this variable were replaced with measures taken from the Common Core of Data. The effect of this 
replacement on the confidentiality of the data was examined as part of the confidentiality analyses 
described earlier. 


^‘*Free or reduced-price lunch is provided as a composite variable in the public-use data set, and as a continuous variable in the restricted-use 
dataset. State benchmarking restricted-use datasets contain both the categorical and composite forms of the variable. 
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Limited-English Proficient (LEP)/Engiish Language Learners (ELL) (TIMSS 4 & 8/PIRLS 
Schooi Questionnaire) 

Principals were asked to report the percentage of such students and were provided with the following 8 
response categories: 0 percent; 1-5 percent; 6-10 percent; 1 1-25 percent; 26-50 percent; 51-75 percent; 
76-90 percent; and over 90 percent. 


Type of Schooi (TfMSS 4 & 8/PfRLS Schooi Questionnaire) 

Principals were asked to identify their schools using one of the following 10 response categories: (1) 
regular public school; (2) regular public school with magnet program;(3) magnet school or school with 
special program; (4) special education; (5) alternative curriculum; (6) vocational; (7) charter school; (8) 
independent private school; (9) religiously affiliated private school; or ( 1 0) other school. 


Missing Data 


Data derived from the student, school and teacher questionnaires and from the student assessments 
contain missing data in varying amounts. Four sources of missing data are identified: 

1 . Not administered . The respondent was not administered the actual item. He or she had 
no chance to read and answer the question. 

2. Omitted . The respondent had a chance to answer the question but did not do so. This 
code also was used for responses that were not interpretable. 

3. Logically not applicable . The respondent answered a preceding filter question in a 
way that made the following dependent questions not applicable to him or her. 

4. Not reached (only used in the achievement files). This code indicates those items not 
reached by the students due to a lack of time. 

SAS and SPSS control code for all the data files include the code for handling/converting missing data. 
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Imputation 


No imputation for missing values was undertaken. However, missing data on the measure of school 
poverty (proportion of students eligible for free or reduced-price lunch) reported by schools was replaced 
as described above. 


5.1 0 Data Files Available From the Website of the National Center 

for Education Statistics 


The data files containing the U.S. national data for TIMSS can be downloaded from the NCES website at 
http://nces.ed.gov/timss/datafiles.asp . The data files containing the U.S. national data for PIRLS can be 
downloaded from the NCES website at http://nces.ed.gov/surveys/pirls/datafiles.asp . These national files 
are in ASCII format and are named as indicated in exhibit 5-5. SAS and SPSS codes for reading these 
data files can also be downloaded from the NCES website. 


Exhibit 5-5. U.S. national data file names 


Contents 

File name 

Grade 4 TIMSS 

Student Background 

T4_STUDENT11.DAT 

Student Achievement 

T4 ACHIEVE11.DAT 

Teacher Background 

T4_TEACHER11.DAT 

Student-Teacher Link 

T4 STD TCH LINK11.DAT 

School Background 

T4_SCH00L11.DAT 

Grade 4 PIRLS 

Student Background 

P4_STUDENT11.DAT 

Student Achievement 

P4_ACHIEVE11.DAT 

Teacher Background 

P4_TEACHER11.DAT 

Student-Teacher Link 

P4 STD TCH LINK11.DAT 

School Background 

P4_SCH00L11.DAT 

Grade 8 TIMSS 

Student Background 

T8_STUDENT11.DAT 

Student Achievement 

T8_ACHIEVE11.DAT 

Math Teacher Background 

T8_MTEACHER1 1 .DAT 

Science Teacher Background 

T8_STEACHER11.DAT 

Student-Teacher Link 

T8_STD_TCH_LINK1 1 .DAT 

School Background 

T8 SCH00L11.DAT 


SOURCE: National Center for Education Statistics, U.S. Department of Education (2013). 


■^The public-use data set and restricted-use data CD contain a quick guide that explains how to use ASCII files with syntax files to create SPSS or 
SAS datasets. 
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5.11 


Merging TIMSS/PIRLS 2011 Data Files 


In preparing TIMSS 2011 or PIRLS 2011 data for analysis it may be neeessary to merge two (or more) of 
the data files named in exhibit 5-5. Not every analysis will require merging of files, however. For 
example, analyses looking at the relationship between student background and achievement can be done 
using the student background file alone. However, analyses that wish to examine the relationships 
between school and/or teacher and/or student characteristics and student achievement will require that 
files be merged. Standard merging procedures as implemented in SPSS, SAS or Stata can be applied. 
Examples are provided below along with illustrative SAS and SPSS code. (These various merges are 
facilitated for SPSS users who choose to work with the lEA International Database Analyzer [lEA IDB 
Analyzer] described below.) 

The merging procedures illustrated below follow the same pattern as previous TIMSS studies (see Foy et 
ah, 2013) and are illustrated with TIMSS Grade 8 data. The user will need to replace file names in order 
to run the merges for TIMSS Grade 4 or PIRES. 


Merging Student and School Data 

If the intent is to disaggregate school data across students the school-level data are merged to the student 
file using IDSCHOOE. The disaggregated data can be analyzed at the student level using the student-level 
weight TOTWGT. Exhibits 5-6 and 5-7 provide examples of how to merge the student and school data 
using SAS and SPSS. Additional examples are provided in chapters 2 and 3 in the TIMSS 2011 User 
Guide for the International Database (Foy et ah, 2013; available at 

http://timssandpirls.bc.edu/timss201 1/intemational-database.html) and PIRES 2011 User Guide for the 
International Database (Foy et. ah, 2013; available at http://timssandpirls.bc.edu/pirls20 1 1/international- 
database.html) . 
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Exhibit 5-6. Illustrative SAS code for merging U.S. TIMSS eighth-grade student and school data 


libname bM5 "C:\T1MSS201 l\Data"; 

data SCHOOL; 

setbM5.T8_SCHOOLll; 

proc sort data= SCHOOL; 

by IDSCHOOL; 

data STUDENT; 

set bM5.T8_STUDENTl 1; 

proc sort data= STUDENT ; 

by IDSCHOOL; 

data bMS.MERGEl; 

merge STUDENT SCHOOL; 

by IDSCHOOL; 

run; 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 

International Mathematics and Science Study (TIMSS), 2011. 


This example creates a temporary SAS dataset (SCHOOL) using the permanent school dataset 
bM5.T8_SCHOOLl 1. It then sorts the school data by school ID (IDSCHOOL). A similar procedure is 
used for the student file (STUDENT), which is also sorted by the school ID using the permanent student 
dataset bM5.T8_STUDENTl 1. The final dataset is a permanent dataset called bMS.MERGEl containing 
the merged file from SCHOOL and STUDENT using IDSCHOOL as the merge variable. 


The SPSS example shown in exhibit 5-7 works in a similar way. SPSS uses a file containing the school 
variables (T8_SCHOOLl l.SAV) and sorts the cases by IDSCHOOL. The same procedure is used for the 
student dataset, T8_STUDENT1 l.SAV. The “match files” command merges the two files, and the final, 
merged output file is saved as MERGE l.SAV. 

Exhibit 5-7. Illustrative SPSS code for merging U.S. TIMSS eighth-grade student and school data 

get file = "C:\TIMSS201 l\Data\T8_SCHOOLl l.SAV". 
sort cases by IDSCHOOL. 

save outfile = 'C:\T1MSS201 l\Data\SCHOOL.SAV. 
get file = "C:\T1MSS201 l\Data\T8_STUDENTl l.SAV". 
sort cases by IDSCHOOL. 

save outfile = 'C:\T1MSS201 l\Data\STUDENT.SAV. 
match files 

/ file= 'C:\T1MSS201 l\Data\STUDENT.SAV 
/ table= 'C:\T1MSS201 l\Data\SCHOOL.SAV 
/by IDSCHOOL. 

save outfile = "C:\T1MSS201 l\Data\MERGEl.SAV". 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Merging Student and Teacher Data 


In the United States, the student sample was based on intaet classrooms. The teachers of the students 
selected in this way are not a sample of teachers and should be seen as the “teachers of the sampled 
students.” To maintain this linkage merges of teacher and student data must use the student-teacher li nk 
file (T8_STD_TCH_LINK1 l.DAT) which also contains the appropriate teacher sampling weights. 
Exhibits 5-8 and 5-9 provide illustrative code for merging the student and the mathematics teacher files in 
SAS and SPSS. The sample code provided illustrates the merge involving the eighth-grade mathematics 
teacher file with student file. The same logic applies to a merge involving the science teacher file 
(T8STEACHER1 l.DAT). Additional examples of how to merge the student and teacher files using SAS 
and SPSS are provided in chapters 2 and 3 of the TIMSS 2011 User Guide for the International Database 
(Foy et ah, 2013; available at http://timssandpirls.bc.edu/timss201 1/intemational-database.html) and 
PIRLS 2011 User Guide for the International Database (Foy et ah, 2013; available at 
http://timssandpirls.bc.edu/pirls201 1/intemational-database.html) . 

Exhibit 5-8. Illustrative SAS code for merging U.S. TIMSS eighth-grade student and mathematics 
teacher data 

Eibname bM5 "C:\T1MSS201 l\Data"; 
data TEACHER; 
set bM5.T8_MTEACHERl 1 ; 
proc sort data= TEACHER; 
by IDTEACH IDEINK; 
data STDTCH; 

set bM5.T8_STD_TCH_ENKl 1 ; 

proc sort data= STDTCH; 

by IDTEACH IDEINK; 

data TEACHMRG; 

merge TEACHER STDTCH; 

by IDTEACH IDEINK; 

ifMATWGT>0; 

proc sort data = TEACHMRG; 

by IDSTUD; 

data STUDENT; 

set bM5.T8_STUDENTl 1; 

proc sort data = STUDENT; 

by IDSTUD; 

data bM5.MERGE2; 

merge STUDENT TEACHMRG; 

by IDSTUD; 

run; 

SOURCE: Intemational Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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In the SAS example, the program creates a temporary SAS dataset (TEACHER) using the permanent 
mathematics teacher file, bM5.T8_MTEACHERl 1. It then sorts the teacher data by the teacher ID 
(IDTEACH) and the link ID (IDLINK). A similar procedure is used for the student-teacher link file 
(STDTCH), using the permanent file (bM5.T8_STD_TCH_EINKl 1) which is also sorted by the teacher 
ID and the link ID. The weight variable for mathematics teachers (MATWGT) is used as a selection 
variable because mathematics teachers have been selected. The result is a merged file called 
bMS.TEACHMRG with disaggregated teacher data. This file is merged with the student file 
(STUDENT). The final dataset is a permanent dataset called bM5.MERGE2 that contains the merged file 
from TEACHMRG and STUDENT using IDSTUD as the merge variable. 


The SPSS student-teacher merge in Exhibit 5-9 uses a file containing the teacher variables 
(T8 MTEACHER1 l.SAV) and sorts the cases by IDTEACH and IDLINK. The file is then saved as 
TEACHER. The same procedure is used for the student-teacher linkage dataset 
T8_STD_TCH_LINK1 l.SAV. The “match files” command merges the two files by the ID variables 
IDTEACH and IDLINK, and the merged output file is saved as TEACHMRG. To include the student 
data, the student file is selected (T8 STUDENT1 l.SAV), sorted by IDSTUD and saved as STUDENT. 
This file is merged with TEACHMRG using IDSTUD to create the final file MERGE2.SAV containing 
both teacher and student variables. 


Exhibit 5-9. Illustrative SPSS code for merging U.S. TIMSS eighth-grade student and mathematics 
teacher data 

get file = "C:\TIMSS201 l\Data\T8_MTEACHERl l.SAV". 

sort cases by IDTEACH IDLINK . 

save outfile = 'C:\TIMSS201 l\Data\TEACHER.SAV'.. 

get file = "C:\TIMSS2011\Data\T8_STD_TCH_LNKll.SAV". 

select if MATWGT > 0. 

sort cases by IDTEACH IDLINK. 

save outfile= 'C:\TIMSS2011\Data\STDTCH.SAV.match files 
/ file='C:\TIMSS201 l\Data\STDTCH.SAV 
/ table='C:\TIMSS201 l\Data\TEACHER.SAV 
/ by IDTEACH IDLINK. 
sort cases by IDSTUD. 

save outfile = 'C:\TIMSS2011\Data\TEACHMRG.SAV'.. 
get file = "C:\TIMSS2011\Data\T8_STUDENTll.SAV". 
sort cases by IDSTUD. 

save outfile= 'C:\TIMSS2011\Data\STUDENT.SAV. 
match files 

/ file= 'C:\TIMSS2011\Data\TEACHMRG.SAV 
/ table= 'C:\TIMSS201 l\Data\STUDENT.SAV 
/ by IDSTUD. 

save outfile = "C:\TIMSS2011\Data\MERGE2.SAV". 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in International 
Mathematics and Science Study (TIMSS), 2011. 
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Merging Student, School, and Teacher Data 


In merging student, teacher, and school data together to form a single dataset, the procedures from 
sections 5.1 1.1 and 5.1 1.2 are combined. Exhibits 5-10 and 5-1 1 show illustrative SAS and SPSS code 
designed to achieve this three-way merge. As in the previous examples, the sample code is based on 
eighth-grade mathematics teacher file. A merge involving the eighth-grade science teacher file follows the 
same logic. 

This example uses the same merging steps as with the previous school and teacher examples (MERGE 1 
and MERGE2), then merges the output files by the student id, IDSTUD, into a final file 
bM5.MERGEAEE containing linked student, school, and teacher data at the student level. 


Exhibit 5-10. Illustrative SAS code for merging U.S. TIMSS eighth-grade school, mathematics teacher 
and student data 

libname bM5 "C:\TIMSS2011\Data"; 

data SCHOOL; 

setbM5.T8_SCHOOLll; 

proc sort data= SCHOOL; 

by ID SCHOOL; 

data STUDENT; 

setbM5.T8_STUDENTll; 

proc sort data= STUDENT ; 

by ID SCHOOL; 

data MERGE I ; 

merge STUDENT SCHOOL; 

by ID SCHOOL; 

proc sort data=MERGE I ; 

by IDSTUD; 

data TEACHER ; 

set bM5.T8_MTEACHERI I; 

proc sort data= TEACHER; 

by IDTEACH IDLINK ; 

data STDTCH ; 

set bM5.T8_STD_TCH_LNKI I; 

proc sort data= STDTCH; 

by IDTEACH IDLINK; 

data MERGE2; 

merge STDTCH TEACHER; 

by IDTEACH IDLINK; 

ifMATWGT>0; 

proc sort data = MERGE2; 

by IDSTUD; 

data bM5.MERGEALL; 

merge MERGEI MERGE2; 

by IDSTUD; 

run; 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in International 

Mathematics and Science Study (TIMSS), 2011. 
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In the SPSS example shown in exhibit 5-1 1 the student and school data are first sorted by IDSCHOOL 
and then merged. The procedure followed for combining student and teacher data in Exhibit 5-9 is used 
again. Then the saved student-school and student-teacher fdes are merged by IDSTUD, and a final dataset 
MERGEALL.SAV is saved. 


Exhibit 5-11. Illustrative SPSS code for merging U.S. TIMSS eighth-grade school, mathematics teacher 
and student data 


get file = "C:\T1MSS201 l\Data\T8_SCHOOLl l.SAV". 
sort cases by IDSCHOOL. 

save outfile = 'C:\TIMSS201 l\Data\SCHOOL.SAV. 
get file = "C:\T1MSS201 l\Data\T8_STUDENTl l.SAV". 
sort cases by IDSCHOOL. 

save outfile = 'C:\T1MSS201 l\Data\STUDENT.SAV. 
match files 

/ file= 'C:\T1MSS201 l\Data\STUDENT.SAV 
/ table= 'C:\T1MSS201 l\Data\SCHOOL.SAV 
/by IDSCHOOL. 

save outfile = "C:\T1MSS201 l\Data\MERGEl.SAV" . 

get file = "C:\T1MSS201 l\Data\T8_MTEACHERl l.SAV". 

sort cases by IDTEACH IDLINK . 

save outfile = 'C:\T1MSS201 l\Data\TEACHER.SAV. 

get file = "C:\T1MSS201 l\Data\T8_STD_TCH_LNKl 1 .SAV". 

select if MATWGT > 0 . 

sort cases by IDTEACH IDLINK. 

save outfile = 'C:\T1MSS201 l\Data\STDTCH.SAV. 

match files 

/ file= 'C:\T1MSS201 l\Data\STDTCH. SAV 
/ table= 'C:\T1MSS201 l\Data\TEACHER. SAV 
/ by IDTEACH IDLINK. 

save outfile = "C:\T1MSS201 l\Data\MERGE2.SAV". 

Get file = "C:\TIMSS201 l\Data\MERGEl. SAV". 

Sort cases by IDSTUD. 

save outfile = 'C:\T1MSS201 l\Data\MERGEl.SAV.Get file = 

"C:\T1MSS201 l\Data\MERGE2.SAV". 

Sort cases by IDSTUD. 

save outfile = 'C:\T1MSS201 l\Data\MERGE2'.match files 
/ file= 'C:\T1MSS201 l\Data\MERGE2.SAV 
/ table= 'C:\T1MSS201 l\Data\MERGE l.SAV 
/by IDSTUD. 

save outfile = "C:\TIMSS201 l\Data\MERGEALL.SAV". 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Merging TIMSS or PIRLS 2011 Data With Restricted-Use Data 


Users who have been granted a license to use the restricted-use TIMSS or PIRLS 2011 data will receive a 
restricted-use CD-ROM which contains an additional link file that provides a way to merge TIMSS or 
PIRLS data with school data from the Common Core of Data (CCD) and the Private School Survey 
(PSS). The NCESSCH (the NCES unique public school identification code) from the TIMSS or PIRLS 
file is used to merge with NCESSCH from the CCD file. The PPIN (the private school’s unique 
identification number) from the TIMSS or PIRLS file is used to merge with the PPIN from the PSS file. 
Illustrative SAS and SPSS code is provided in Exhibits 5-12 and 5-13. 

The code in question provides for a link between the TIMSS school data and the CCD/PSS data by 
school. Further merging to other TIMSS files (student, school, teacher) can be conducted using the 
IDSCHOOL for merging as has been shown in earlier examples. 
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Exhibit 5-12. Illustrative SAS code for merging U.S. TIMSS eighth-grade school data with restricted- 
use (CCD and PSS) data 

libname bM5 "C:\T1MSS201 l\Data" ; 
data SCHOOL ; 

set bM5.T8_RESTRlCTED_USEl 1; 

proc sort data= SCHOOL; 

by NCESSCH ; 

data CCD ; 

set bMS.CCD; 

proc sort data= CCD; 

by NCESSCH; 

data MERGE 1 ; 

merge CCD(IN=IN1) SCHOOL (IN=IN2); 
by NCESSCH; 

IF 1N2; 
run; 

/* User can Merge in PSS data to the file containing CCD and TIMSS previously merged 
data */ 

Data SCHOOL2; 

Set MERGE 1; 

proc sort data= SCHOOL2; 

by PPIN ; 

data PSS ; 

set bMS.PSS; 

proc sort data= PSS; 

by PPIN; 

data MERGE2 ; 

merge PSS(IN=IN1) SCHOOL2 (IN=IN2); 
by PPIN; 

IF 1N2; 
run; 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


SPSS code designed to provide this same link between the TIMSS school data and the CCD/PSS data by 
school is provided in exhibit 5-13. Further merging to other TIMSS files (student, school, teacher) can be 
conducted using the IDSCHOOL for merging as has been shown in earlier examples. 
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Exhibit 5-13. Illustrative SPSS code for merging U.S. TIMSS eighth-grade school data with restricted 
(CCD and PSS) data 

get file = "C:\T1MSS201 l\Data\T8_RESTRICTED_USEl l.SAV". 
sort cases by NCESSCH. 

save outfile = 'C:\T1MSS201 l\Data\SCHOOL.SAV'.get file = 

"C:\T1MSS201 l\Data\CCD.SAV". 

sort cases by NCESSCH. 

save outfile = 'C:\T1MSS201 l\Data\CCD.SAV. 

match files 

/ file= 'C:\T1MSS201 l\Data\CCD.SAV 
/ table= 'C:\T1MSS201 l\Data\SCHOOL.SAV 
/by NCESSCH. 

save outfile = "C:\TIMSS201 l\Data\MERGEl.SAV". 

* Merge PSS to the Combined TIMSS/CCD school-level file 

get file = "C:\TIMSS201 l\Data\MERGEl.SAV". 
sort cases by PPIN. 

save outfile = 'C:\T1MSS201 l\Data\SCHOOL2.SAV. 
get file = "C:\T1MSS201 l\Data\PSS.SAV". 
sort cases by PPIN. 

save outfile = 'C:\T1MSS201 l\Data\PSS.SAV. 
match files 

/ file= 'C:\T1MSS201 l\Data\PSS.SAV 
/ table= 'C:\T1MSS201 l\Data\SCHOOL2.SAV 
/ by PPIN. 

save outfile = "C:\T1MSS201 l\Data\MERGE2.SAV" . 

SOURCE: International Association for the Evaluation of Education Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 201 E 


5.12 Some Notes on Analyzing the TIMSS and PIRLS 2011 Data 

The design of TIMSS and PIRLS raises three special considerations for the analysis of TIMSS and PIRLS 
data. First, the assessment design necessitates the use of five plausible values rather than a single score 
for each of the various measures of mathematics, science, and reading achievement. Second, since the 
sampling design is not a simple random sample in which each student had an equal probability of 
selection, sampling weights need to be applied to generate unbiased estimates of population parameters. 
Third, the complex sampling design also means that the calculation of the standard errors of the various 
statistics generated requires special procedures. 
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Plausible Values 


As noted earlier, the assessment design was based on Balanced Incomplete Block (BIB) spiraling of 
assessment items to increase content-area coverage without a concomitant increase in the assessment time 
demanded of students. Each student completed only a subset of the total pool of assessment items, with 
the resulting data containing missing values for other items in the pool but not in the subset administered 
to the student. The trade-off for increased coverage through BIB spiraling is increased measurement error 
in the scores available for each student. This is accommodated through the estimation of (five) plausible 
values for each student rather than a single (unreliable) point estimate. Plausible values are random draws 
from the estimated distribution of a student’s achievement. A detailed description of the TIMSS or PIRLS 
2011 scaling can be found in Methods and Procedures in TIMSS and PIRLS 2011 (Martin & Mullis, 
2011 ). 

What this means for those analyses of TIMSS and PIRLS data that include achievement measures is that 
the analyses need to be done five times and the results averaged. For example, if one was regressing 
mathematics achievement on a number of family and school attributes, it would be necessary to estimate 
this equation five times and then average each set of five parameter estimates. It would not be legitimate 
to take the mean of the five plausible values in the first instance and then regress this mean on a number 
of family and school attributes. 


Estimating Sampling Variance 

The complex sampling design used in TIMSS and PIRLS 2011 complicates the task of computing 
standard errors. Most standard analysis software systems such as SAS and SPSS provide estimates based 
on the assumption of a simple random sample. Given the TIMSS and PIRLS sampling design, such 
standard errors will underestimate the true standard errors. TIMSS and PIRLS adopt the jackknife 
repeated replication (JRR) technique because it is computationally straightforward and provides 
approximately unbiased estimates of the sampling errors of means, totals, and percentages. The variables 
necessary for these JRR procedures are included as part of the TIMSS and PIRLS 2011 data files: 
JKZONE, the sampling zone (stratum) of the student’s school; and JKREP, the sampling replicate 
(primary sampling unit) of the student’s school. 

There are, however, several options for estimating sampling errors that avoid the assumption of simple 
random sampling. The SPSS-linked International Database (IDB) Analyzer software was designed 
specifically by International Association for the Evaluation of Educational Achievement (lEA) for 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


163 



analyzing TIMSS and PIRLS international data files. This software is freely available from the lEA 
website at http://www.iea.nl/data.html . 


Speeial-use software is also available for estimating the standard errors of statisties generated from 
complex sampling designs. Among the packages available are AM, available from the American Institutes 
for Research at http://am.air.org/about2.asp : WesVar, available from Westat at 
http://www.westat.com/Westat/expertise/information systems/WesVar/wesvar downloads. cfm, - and 
SUDAAN, available from Research Triangle Institute at http://www.rti.org/sudaan . Some software 
packages provide for these capabilities as well. 

In addition, SAS macros suitable for this purpose are available as part of the TIMSS 2011 User Guide for 
the International Database, as well as in the PIRLS 2011 User Guide for the International Database 
(Toy, Arora, & Stanco, 2013; Toy & Drucker, 2013). See also work by Stapleton (2006, 2008), which 
suggests procedures that can be used to generate appropriate standard errors for statistics generated by 
structural equation modeling techniques. 


The lEA International Database (IDB) Analyzer and International Data Explorer 
(IDE) 

As described in section 5.12.2, the IDB Analyzer was developed by the lEA Data Processing and 
Research Center (lEA DPC) as a plug-in for SPSS and can only be used in conjunction with SPSS. It is 
not a stand-alone analysis system. 

The IDB Analyzer enables users to combine SPSS data files and conduct analyses using SPSS without 
actually writing programming code. The IDB Analyzer generates SPSS syntax that takes into account 
information from the sampling design in the computation of statistics and their standard errors. In 
addition, the generated SPSS syntax makes appropriate use of plausible values for calculating estimates of 
achievement scores and their standard errors, combining both sampling variance and imputation variance. 

The IDB Analyzer consists of two modules — a merge module and an analysis module. The merge module 
is used to create analysis datasets by combining data files of different types and from different countries 
and selecting subsets of variables for analysis. The analysis module provides procedures for computing 
various statistics and their standard errors. All statistical procedures offered within the analysis module of 
the IDB Analyzer make appropriate use of sampling weights, and standard errors are computed using the 
jackknife repeated replication (JRR) method. Percentages, means, regressions, and correlations may be 
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specified with or without achievement scores. When achievement scores are used, the analyses are 
performed five times — once for each plausible value — and the results are aggregated to produce accurate 
estimates of achievement and standard errors that incorporate both sampling and imputation errors. 

The use of the IDB Analyzer is described in detail with worked examples in chapter 2 of both the TIMSS 
2011 User Guide for the International Database and the PIRLS 2011 User Guide for the International 
Database (Foy, Arora, & Stanco, 2013; Foy & Drucker, 2013). Readers intending to use this user-friendly 
software are urged to read this user guide in detail. 

In addition to IDB Analyzer for basic analysis and exploration of TIMSS and PIRLS data, NCES has 
developed a relatively simple, interactive online data-analysis tool: the International Data Explorer (IDE), 
can be found at http://nces.ed.gov/survevs/intemational/ide/ . The IDE allows users to analyze all the 
international variables for all participating education systems and the U.S. -specific variables; however, it 
does not include U.S. restricted-use data. The IDE does not require SPSS for analyzing the data. It 
provides users with the capabilities to create statistical tables and charts of TIMSS and PIRLS data across 
countries and years on the website. This tool allows users to point and click in a self-contained module, 
unlike the IDB Analyzer software that must be used in conjunction with SPSS. Also unlike the IDB 
Analyzer, the IDE does not provide access to data files for merging, transforming, or otherwise 
manipulating data. This tool reports averages for subject by selected variables and exports reports in 
HTME, Excel, Word, or PDF. 


SAS Programs and Macros 

The TIMSS 2011 User Guide for the International Database and the PIRLS 2011 User Guide for the 
International Database also provide assistance for those investigators who wish to conduct their analyses 
using SAS. The user guide includes a number of SAS programs needed to process the SAS data files, 
compute survey results, and carry out example analyses. These are described in detail with worked 
examples in chapter 3 of the user guide. Readers intending to use SAS for their analyses are urged to read 
this chapter in detail. 
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The following SAS programs and macros are available: 


■ CONVERT. SAS: is used to convert SAS Export files into SAS data files. 

■ ASASCRM4.SAS, BSASCRM4.SAS: is used to convert the response codes on the 
achievement items to their corresponding score levels. 

■ JOIN. SAS: combines files of the same type from more than one education system. 

■ JACKGEN.SAS: is an SAS macro used to compute weighted percentages of students 
within defined subgroups, along with their means on a specified continuous variable. 
This macro generates replicate weights and computes standard errors using the 
jackknife repeated replication (JRR) methodology. The analysis variable can be any 
continuous variable. (When computing mean achievement scores with plausible 
values, the macro JACKPV.SAS should be used.) 

■ JACKPV.SAS: is an SAS macro used to compute weighted percentages of students 
within defined subgroups, along with their mean achievement on a scale using the 
available plausible values. This macro generates replicate weights and computes 
standard errors using the jackknife repeated replication (JRR) and multiple imputation 
methodologies. This macro should be used when achievement plausible values are 
used in an analysis. 

■ JACKREG.SAS: is a SAS macro used to compute weighted regression coefficients 
and their standard errors within defined subgroups. This macro can be used with any 
analysis variable, but is not appropriate for analyzing achievement with plausible 
values. 

■ JACKREGP.SAS: is a SAS macro program used to compute weighted regression 
coefficients and their standard errors within defined subgroups when using 
achievement plausible values as the dependent variable. 


Special Considerations in Using the Teacher Data 

The teachers in the TIMSS and PIRES 2011 international databases are the teachers of nationally 
representative samples of students and are not representative samples of teachers in the participating 
countries. As a result, analyses with teacher data should be made with students as the units of analysis and 
reported in ter ms of students who are taught by teachers with a particular attribute. 

When analyzing teacher data, it is first necessary to link the students to their respective teachers. The 
student-teacher linkage data files (AST/BST) were created for this purpose. Since student achievement 
scores (plausible values), jackknife replication information, and teacher weighting variables are found in 
the student-teacher linkage data files, it is only necessary to merge the teacher background data files with 
the student-teacher linkage data files. For analyses linking teacher variables to student background 
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variables, it is also necessary to merge the student background data fdes with the teacher background data 
fdes after having been combined with the student-teacher linkage data files. 


In general, to perform analyses using the teacher background data files, follow the steps below. 

1 . Identify the variables of interest in the teacher background data files and note any 
specific national adaptations to the variables. 

2. Retrieve the relevant variables from the teacher background data fdes, including 
analysis variables, classification variables, identification variables (IDCNTRY, 
IDTEACH, and IDLINK), and any other variables used in the selection of cases. 

3. Retrieve the relevant variables from the student-teacher linkage data fdes, including 
plausible values of achievement, classification variables, identification variables 
(IDCNTRY, IDSTUD, IDTEACH, and IDLINK), sampling (JKZONE and JKREP) 
and weighting (MATWGT, SCIWGT, or TCHWGT) variables, and any other 
variables used in the selection of cases. 

4. Merge the teacher background data fdes with the student-teacher linkage data fdes 
using the variables IDCNTRY, IDTEACH, and IDLINK. 

5. If student background variables also are needed, merge the student background data 
fdes with the merged student-teacher data fdes from the previous step using the 
variables IDCNTRY and IDSTUD. 

One further point to note: fourth-grade teachers were given a single questionnaire with mathematics and 
science sections or a reading section as pertinent to PIRLS. If a teacher taught only mathematics or only 
science to the TIMSS fourth-graders, that teacher would complete only one of these sections. In 
frequency distributions of the variables in these sections, teachers who did not answer the questions for 
this reason are shown as “not administered.” 


Special Considerations in Using the School Data 


In general, to perform analyses using the school background data fdes, follow the steps below. 

1 . Identify the variables of interest in the school and student background data fdes and 
note any specific national adaptations to the variables. 

2. Retrieve the relevant variables from the school background data files, including 
analysis variables, classification variables, identification variables (IDCNTRY and 
IDSCHOOL), and any other variables used in the selection of cases. 
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3. Retrieve the relevant variables from the student background data files, including 
plausible values of achievement, classification variables, identification variables 
(IDCNTRY and IDSCHOOL), sampling (JKZONE and JKREP) and weighting 
(TOTWGT) variables, and any other variables used in the selection of cases. 

4. Merge the school background data files with the student background data files using 
the variables IDCNTRY and IDSCHOOL. 
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Massachusetts TIMSS eighth-grade school sampling frame and sample, 
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Table A-1 . Number and percentage of students and schools included in the Florida TIMSS-PIRLS 

fourth-grade school sampling frame and sample, by poverty level, locale, and race/ethnicity 
status: 2011 


Poverty 

level 

Locale 

Race/ethnicity status 

Frame 


Sample 


Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



204,215 

100 

81 

100 

High 

City 

1 5 percent or above 

28,918 

14.2 

11 

13.6 

High 

City 

Below 15 percent 

65 

0.0 

0 

0.0 

High 

Suburb 

1 5 percent or above 

55,745 

27.3 

23 

28.4 

High 

Suburb 

Below 15 percent 

1,060 

0.5 

0 

0.0 

High 

Town 

1 5 percent or above 

7,673 

3.8 

4 

4.9 

High 

Town 

Below 15 percent 

302 

0.1 

0 

0.0 

High 

Rural 

1 5 percent or above 

12,758 

6.2 

3 

3.7 

High 

Rural 

Below 1 5 percent 

2,951 

1.4 

1 

1.2 

Low 

City 

1 5 percent or above 

17,799 

8.7 

6 

7.4 

Low 

City 

Below 1 5 percent 

1,988 

1.0 

1 

1.2 

Low 

Suburb 

1 5 percent or above 

40,423 

19.8 

16 

19.8 

Low 

Suburb 

Below 1 5 percent 

11,287 

5.5 

4 

4.9 

Low 

Town 

1 5 percent or above 

2,501 

1.2 

2 

2.5 

Low 

Town 

Below 1 5 percent 

1,370 

0.7 

1 

1.2 

Low 

Rural 

1 5 percent or above 

13,751 

6.7 

5 

6.2 

Low 

Rural 

Below 1 5 percent 

5,624 

2.8 

4 

4.9 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Flispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Flispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), Progress in International Reading Literacy Study 
(PIRLS), 2011. 
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Table A-2. Number and percentage of students and schools included in the North Carolina TIMSS 

fourth-grade school sampling frame and sample, by poverty level, locale, and race/ethnicity 
status: 2011 


Poverty 

level 

Locale 

Race/ethnicity status 

Frame 


Sample 


Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



116,717 

100 

49 

100 

High 

City 

1 5 percent or above 

11,521 

9.9 

5 

10.2 

High 

City 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Suburb 

1 5 percent or above 

3,833 

3.3 

1 

2.0 

High 

Suburb 

Below 1 5 percent 

807 

0.7 

1 

2.0 

High 

Town 

1 5 percent or above 

7,462 

6.4 

3 

6.1 

High 

Town 

Below 1 5 percent 

99 

0.1 

0 

0.0 

High 

Rural 

1 5 percent or above 

13,411 

11.5 

6 

12.2 

High 

Rural 

Below 1 5 percent 

4,532 

3.9 

2 

4.1 

Low 

City 

1 5 percent or above 

17,869 

15.3 

7 

14.3 

Low 

City 

Below 1 5 percent 

1,754 

1.5 

1 

2.0 

Low 

Suburb 

1 5 percent or above 

10,089 

8.6 

4 

8.2 

Low 

Suburb 

Below 1 5 percent 

5,335 

4.6 

2 

4.1 

Low 

Town 

1 5 percent or above 

5,473 

4.7 

2 

4.1 

Low 

Town 

Below 1 5 percent 

1,603 

1.4 

1 

2.0 

Low 

Rural 

1 5 percent or above 

21,644 

18.5 

10 

20.4 

Low 

Rural 

Below 1 5 percent 

11,285 

9.7 

4 

8.2 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined 
as having 50 percent or more of the students eligible for participation in the National School Lunch 
Program (NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP 
data were available for private schools, all private schools are categorized as “low.” All private schools 
are categorized as “low.” For definitions of these urban-centric locales, see 

http://nces.ed.gov/survevs/urbaned/deFinitions.asp . Race/ethnicity status refers to the percentage of Black, 
Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native students. Black includes 
African American and Hispanic includes Latino. Racial categories exclude Hispanic origin. Asian and 
Pacific Islander are combined into a single category due to the fact that that the frame was developed 
using the previous NCES guidelines for combining Asian and Pacific Islanders, while the U.S. national 
report was developed under new guidelines that call for reporting out Asian separately when possible. The 
measure of size is defined as the estimated number of students enrolled in the target grade with a 
minimum of 5 students per school. These differ slightly from the estimated student sample size reported 
in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of students in the sampled 
schools with no minimum per school. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-3. 


Number and percentage of students and schools included in the Alabama TIMSS eighth- 
grade school sampling frame and sample, by poverty level, locale, and race/ethnicity status: 


2011 


Poverty 

level 

Locale 

Race/ethnicity status 

Frame 


Sample 


Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



58,410 

100 

63 

100 

High 

City 

1 5 percent or above 

8,479 

14.5 

8 

12.7 

High 

City 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Suburb 

1 5 percent or above 

2,953 

5.1 

3 

4.8 

High 

Suburb 

Below 1 5 percent 

199 

0.3 

0 

0.0 

High 

Town 

1 5 percent or above 

4,905 

8.4 

4 

6.3 

High 

Town 

Below 1 5 percent 

526 

0.9 

1 

1.6 

High 

Rural 

1 5 percent or above 

9,690 

16.6 

11 

17.5 

High 

Rural 

Below 1 5 percent 

3,396 

5.8 

3 

4.8 

Low 

City 

1 5 percent or above 

3,859 

6.6 

4 

6.3 

Low 

City 

Below 1 5 percent 

319 

0.5 

1 

1.6 

Low 

Suburb 

1 5 percent or above 

5,738 

9.8 

4 

6.3 

Low 

Suburb 

Below 1 5 percent 

1,924 

3.3 

5 

7.9 

Low 

Town 

1 5 percent or above 

2,559 

4.4 

2 

3.2 

Low 

Town 

Below 1 5 percent 

974 

1.7 

1 

1.6 

Low 

Rural 

1 5 percent or above 

5,338 

9.1 

7 

11.1 

Low 

Rural 

Below 1 5 percent 

7,551 

12.9 

9 

14.3 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no mi nimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-4. 


Number and percentage of students and schools included in the California TIMSS eighth- 
grade school sampling frame and sample, by poverty level, locale, and race/ethnicity status: 


2011 


Poverty 

level 

Locale 

Race/ethnicity status 

Frame 


Sample 


Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



459,628 

100 

94 

100 

High 

City 

1 5 percent or above 

127,809 

27.8 

27 

28.7 

High 

City 

Below 1 5 percent 

760 

0.2 

0 

0.0 

High 

Suburb 

1 5 percent or above 

94,046 

20.5 

20 

21.3 

High 

Suburb 

Below 1 5 percent 

199 

0.0 

0 

0.0 

High 

Town 

1 5 percent or above 

20,087 

4.4 

5 

5.3 

High 

Town 

Below 1 5 percent 

464 

0.1 

0 

0.0 

High 

Rural 

1 5 percent or above 

14,531 

3.2 

4 

4.3 

High 

Rural 

Below 1 5 percent 

541 

0.1 

0 

0.0 

Low 

City 

1 5 percent or above 

57,810 

12.6 

12 

12.8 

Low 

City 

Below 1 5 percent 

13,910 

3.0 

3 

3.2 

Low 

Suburb 

1 5 percent or above 

75,221 

16.4 

14 

14.9 

Low 

Suburb 

Below 1 5 percent 

15,827 

3.4 

2 

2.1 

Low 

Town 

1 5 percent or above 

9,218 

2.0 

2 

2.1 

Low 

Town 

Below 1 5 percent 

1,685 

0.4 

0 

0.0 

Low 

Rural 

1 5 percent or above 

23,414 

5.1 

5 

5.3 

Low 

Rural 

Below 1 5 percent 

4,106 

0.9 

0 

0.0 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-5. 


Number and percentage of students and schools included in the Colorado TIMSS eighth- 
grade school sampling frame and sample, by poverty level, locale, and race/ethnicity status: 


2011 


Poverty 

level 

Locale 


Frame 


Sample 


Race/ethnicity status 

Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



58,010 

100 

60 

100 

High 

City 

1 5 percent or above 

9,491 

16.4 

8 

13.3 

High 

City 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Suburb 

1 5 percent or above 

3,465 

6.0 

5 

8.3 

High 

Suburb 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Town 

1 5 percent or above 

1,941 

3.3 

2 

3.3 

High 

Town 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Rural 

1 5 percent or above 

1,228 

2.1 

1 

1.7 

High 

Rural 

Below 1 5 percent 

119 

0.2 

0 

0.0 

Low 

City 

1 5 percent or above 

6,788 

11.7 

9 

15.0 

Low 

City 

Below 1 5 percent 

2,874 

5.0 

3 

5.0 

Low 

Suburb 

1 5 percent or above 

7,689 

13.3 

7 

11.7 

Low 

Suburb 

Below 1 5 percent 

7,817 

13.5 

8 

13.3 

Low 

Town 

1 5 percent or above 

2,123 

3.7 

2 

3.3 

Low 

Town 

Below 1 5 percent 

1,647 

2.8 

2 

3.3 

Low 

Rural 

1 5 percent or above 

6,214 

10.7 

8 

13.3 

Low 

Rural 

Below 1 5 percent 

6,614 

11.4 

5 

8.3 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-6. 


Number and percentage of students and schools included in the Connecticut TIMSS eighth- 
grade school sampling frame and sample, by poverty level, locale, and race/ethnicity status: 


2011 


Poverty 

level 

Locale 


Frame 


Sample 


Race/ethnicity status 

Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



42,577 

100 

63 

100 

High 

City 

1 5 percent or above 

4,728 

11.1 

6 

9.5 

High 

City 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Suburb 

1 5 percent or above 

3,063 

7.2 

5 

7.9 

High 

Suburb 

Below 1 5 percent 

327 

0.8 

1 

1.6 

High 

Town 

1 5 percent or above 

240 

0.6 

0 

0.0 

High 

Town 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Rural 

1 5 percent or above 

5 

0.0 

0 

0.0 

High 

Rural 

Below 1 5 percent 

0 

0.0 

0 

0.0 

Low 

City 

1 5 percent or above 

5,486 

12.9 

10 

15.9 

Low 

City 

Below 1 5 percent 

390 

0.9 

0 

0.0 

Low 

Suburb 

1 5 percent or above 

4,365 

10.3 

5 

7.9 

Low 

Suburb 

Below 1 5 percent 

15,948 

37.5 

25 

39.7 

Low 

Town 

1 5 percent or above 

0 

0.0 

0 

0.0 

Low 

Town 

Below 1 5 percent 

1,356 

3.2 

1 

1.6 

Low 

Rural 

1 5 percent or above 

788 

1.9 

1 

1.6 

Low 

Rural 

Below 1 5 percent 

5,881 

13.8 

9 

14.3 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-7. Number and percentage of students and schools included in the Florida TIMSS eighth-grade 
school sampling frame and sample, by poverty level, locale, and race/ethnicity status: 2011 


Poverty 

level 

Locale 


Frame 


Sample 


Race/ethnicity status 

Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



196,879 

100 

65 

100 

High 

City 

1 5 percent or above 

21,179 

10.8 

8 

12.3 

High 

City 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Suburb 

1 5 percent or above 

42,823 

21.8 

14 

21.5 

High 

Suburb 

Below 1 5 percent 

853 

0.4 

1 

1.5 

High 

Town 

1 5 percent or above 

7,525 

3.8 

1 

1.5 

High 

Town 

Below 1 5 percent 

189 

0.1 

0 

0.0 

High 

Rural 

1 5 percent or above 

11,033 

5.6 

4 

6.2 

High 

Rural 

Below 1 5 percent 

1,281 

0.7 

0 

0.0 

Low 

City 

1 5 percent or above 

22,953 

11.7 

12 

18.5 

Low 

City 

Below 1 5 percent 

1,226 

0.6 

0 

0.0 

Low 

Suburb 

1 5 percent or above 

47,056 

23.9 

13 

20.0 

Low 

Suburb 

Below 1 5 percent 

11,063 

5.6 

3 

4.6 

Low 

Town 

1 5 percent or above 

4,385 

2.2 

1 

1.5 

Low 

Town 

Below 1 5 percent 

1,481 

0.8 

1 

1.5 

Low 

Rural 

1 5 percent or above 

18,242 

9.3 

5 

7.7 

Low 

Rural 

Below 1 5 percent 

5,590 

2.8 

2 

3.1 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http://nces.ed.gov/survevs/urbaned/definitions.asp . Race/ethnicity status refers to the 
percentage of Black, Flispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-8. Number and percentage of students and schools included in the Indiana TIMSS eighth-grade 
school sampling frame and sample, by poverty level, locale, and race/ethnicity status: 2011 


Poverty 

level 

Locale 

Race/ethnicity status 

Frame 


Sample 


Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



80,576 

100 

62 

100 

High 

City 

1 5 percent or above 

15,323 

19.0 

13 

21.0 

High 

City 

Below 1 5 percent 

404 

0.5 

1 

1.6 

High 

Suburb 

1 5 percent or above 

2,724 

3.4 

2 

3.2 

High 

Suburb 

Below 1 5 percent 

368 

0.5 

0 

0.0 

High 

Town 

1 5 percent or above 

559 

0.7 

1 

1.6 

High 

Town 

Below 1 5 percent 

1,579 

2.0 

2 

3.2 

High 

Rural 

1 5 percent or above 

1,101 

1.4 

1 

1.6 

High 

Rural 

Below 1 5 percent 

258 

0.3 

0 

0.0 

Low 

City 

1 5 percent or above 

2,537 

3.1 

2 

3.2 

Low 

City 

Below 1 5 percent 

3,448 

4.3 

2 

3.2 

Low 

Suburb 

1 5 percent or above 

6,286 

7.8 

3 

4.8 

Low 

Suburb 

Below 1 5 percent 

10,908 

13.5 

7 

11.3 

Low 

Town 

1 5 percent or above 

873 

1.1 

0 

0.0 

Low 

Town 

Below 1 5 percent 

11,717 

14.5 

11 

17.7 

Low 

Rural 

1 5 percent or above 

277 

0.3 

0 

0.0 

Low 

Rural 

Below 1 5 percent 

22,214 

27.6 

17 

27.4 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http://nces.ed.gov/survevs/urbaned/definitions.asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-9. Number and percentage of students and schools included in the Massachusetts TIMSS 

eighth-grade school sampling frame and sample, by poverty level, locale, and race/ethnicity 
status: 2011 


Poverty 

level 

Locale 


Frame 


Sample 


Race/ethnicity status 

Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



72,953 

100 

58 

100 

High 

City 

1 5 percent or above 

9,779 

13.4 

7 

12.1 

High 

City 

Below 15 percent 

464 

0.6 

1 

1.7 

High 

Suburb 

1 5 percent or above 

8,341 

11.4 

7 

12.1 

High 

Suburb 

Below 1 5 percent 

5 

0.0 

0 

0.0 

High 

Town 

1 5 percent or above 

0 

0.0 

0 

0.0 

High 

Town 

Below 15 percent 

189 

0.3 

0 

0.0 

High 

Rural 

1 5 percent or above 

0 

0.0 

0 

0.0 

High 

Rural 

Below 1 5 percent 

0 

0.0 

0 

0.0 

Low 

City 

1 5 percent or above 

2,515 

3.4 

2 

3.4 

Low 

City 

Below 1 5 percent 

2,279 

3.1 

3 

5.2 

Low 

Suburb 

1 5 percent or above 

6,300 

8.6 

5 

8.6 

Low 

Suburb 

Below 1 5 percent 

32,747 

44.9 

25 

43.1 

Low 

Town 

1 5 percent or above 

566 

0.8 

1 

1.7 

Low 

Town 

Below 1 5 percent 

967 

1.3 

0 

0.0 

Low 

Rural 

1 5 percent or above 

0 

0.0 

0 

0.0 

Low 

Rural 

Below 1 5 percent 

8,801 

12.1 

7 

12.1 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-10. 


Number and percentage of students and schools included in the Minnesota TIMSS eighth- 
grade school sampling frame and sample, by poverty level, locale, and race/ethnicity status: 


2011 


Poverty 

level 

Locale 


Frame 


Sample 


Race/ethnicity status 

Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



62,259 

100 

60 

100 

High 

City 

1 5 percent or above 

5,070 

8.1 

4 

6.7 

High 

City 

Below 1 5 percent 

350 

0.6 

0 

0.0 

High 

Suburb 

1 5 percent or above 

1,396 

2.2 

1 

1.7 

High 

Suburb 

Below 1 5 percent 

12 

0.0 

0 

0.0 

High 

Town 

1 5 percent or above 

341 

0.5 

1 

1.7 

High 

Town 

Below 1 5 percent 

109 

0.2 

0 

0.0 

High 

Rural 

1 5 percent or above 

625 

1.0 

1 

1.7 

High 

Rural 

Below 1 5 percent 

521 

0.8 

0 

0.0 

Low 

City 

1 5 percent or above 

4,217 

6.8 

6 

10.0 

Low 

City 

Below 1 5 percent 

3,110 

5.0 

3 

5.0 

Low 

Suburb 

1 5 percent or above 

8,832 

14.2 

8 

13.3 

Low 

Suburb 

Below 1 5 percent 

9,752 

15.7 

10 

16.7 

Low 

Town 

1 5 percent or above 

2,631 

4.2 

3 

5.0 

Low 

Town 

Below 1 5 percent 

9,516 

15.3 

8 

13.3 

Low 

Rural 

1 5 percent or above 

1,403 

2.3 

1 

1.7 

Low 

Rural 

Below 1 5 percent 

14,374 

23.1 

14 

23.3 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/ definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Table A-1 1 . Number and percentage of students and schools included in the North Carolina TIMSS 
eighth-grade school sampling frame and sample, by poverty level, locale, and 
race/ethnicity status: 201 1 


Poverty 

level 

Locale 

Race/ethnicity status 

Frame 


Sample 


Measure of 
size 

Percent 

Number of 
schools 

Percent 

Total 



110,671 

100 

62 

100 

High 

City 

1 5 percent or above 

8,420 

7.6 

5 

8.1 

High 

City 

Below 1 5 percent 

0 

0.0 

0 

0.0 

High 

Suburb 

1 5 percent or above 

1,646 

1.5 

1 

1.6 

High 

Suburb 

Below 1 5 percent 

291 

0.3 

0 

0.0 

High 

Town 

1 5 percent or above 

6,458 

5.8 

2 

3.2 

High 

Town 

Below 1 5 percent 

546 

0.5 

0 

0.0 

High 

Rural 

1 5 percent or above 

13,462 

12.2 

8 

12.9 

High 

Rural 

Below 1 5 percent 

1,746 

1.6 

2 

3.2 

Low 

City 

1 5 percent or above 

17,283 

15.6 

10 

16.1 

Low 

City 

Below 1 5 percent 

1,298 

1.2 

2 

3.2 

Low 

Suburb 

1 5 percent or above 

11,487 

10.4 

6 

9.7 

Low 

Suburb 

Below 1 5 percent 

4,114 

3.7 

2 

3.2 

Low 

Town 

1 5 percent or above 

6,233 

5.6 

3 

4.8 

Low 

Town 

Below 1 5 percent 

1,128 

1.0 

0 

0.0 

Low 

Rural 

1 5 percent or above 

24,868 

22.5 

14 

22.6 

Low 

Rural 

Below 1 5 percent 

11,691 

10.6 

7 

11.3 


NOTE: Detail may not sum to totals because of rounding. For public schools, “high” poverty is defined as 
having 50 percent or more of the students eligible for participation in the National School Lunch Program 
(NSLP), and “low” poverty is defined as having less than 50 percent eligible. Because no NSLP data were 
available for private schools, all private schools are categorized as “low.” For definitions of these urban- 
centric locales, see http ://nces . ed. go v/surveys/urbaned/definitions . asp . Race/ethnicity status refers to the 
percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude 
Hispanic origin. Asian and Pacific Islander are combined into a single category due to the fact that that 
the frame was developed using the previous NCES guidelines for combining Asian and Pacific Islanders, 
while the U.S. national report was developed under new guidelines that call for reporting out Asian 
separately when possible. The measure of size is defined as the estimated number of students enrolled in 
the target grade with a minimum of 5 students per school. These differ slightly from the estimated student 
sample size reported in Joncas 2012 (Exhibit 1 and 2), which reports the estimate of the number of 
students in the sampled schools with no minimum per school. 

SOURCE: Intemational Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 
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Appendix B. State Participation Rates 
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Table B-1. Number of sampled schools, eighth-grade classrooms, and eighth-grade students in Alabama 
that participated in TIMSS, and their participation rates: 2011 



Number 

Rates 

Unweighted 

Weighted 

Sampled schools 

63 



Excluded and ineligible 

3 



Eligible 

60 



Participated (among original sampled set of schools) 

55 

92 

92 

Substitutes 

0 



Participated (original and substitute schools combined) 

55 

92 

92 

Classrooms in participating schools 




Total 

540 



Excluded 

13 



Eligible 

527 



Sampled 

99 



Participating 

99 

100 

100 

Students in participating schools 




Sampled 

2,414 



Excluded 

87 



Withdrawn 

27 



Eligible 

2,300 



Absent/refusal 

187 



Participating 

2,113 

92 

92 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the Alabama TIMSS grade 8 sample 

The Alabama TIMSS eighth grade sample had a (weighted) student exclusion rate of 4.6 percent based on 
the combination of whole -class and within-class exclusions. lEA standards define this degree of coverage 
of the national target population (95.4 percent) as acceptable. 
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Table B-2. Number of sampled schools, eighth-grade classrooms, and eighth-grade students in 
California that participated in TIMSS, and their participation rates: 2011 



Number 

Rates 

Unweighted 

Weighted 

Schools 




Sampled 

94 



Excluded and ineligible 

1 



Eligible 

93 



Participating 

79 

85 

85 

Substitutes 

3 



Participating (all schools) 

82 

88 

88 

Classrooms in participating schools 




Total 

1,252 



Excluded 

107 



Eligible 

1,145 



Sampled 

118 



Participating 

118 

99 

99 

Students in participating schools 




Sampled 

2,898 



Excluded 

47 



Withdrawn 

52 



Eligible 

2,799 



Absent/refusal 

185 



Participating 

2,614 

93 

94 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the California TIMSS grade 8 sample 

The California TIMSS eighth grade sample had a (weighted) student exclusion rate of 5.6 percent based 
on the combination of whole-class and within-class exclusions. lEA standards define this degree of 
coverage of the national target population (94.4 percent) as acceptable though falling below the desired 
range of 95 percent or better. The tabulations shown in the international reports show California annotated 
to indicate this fact. 
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Table B-3. Number of sampled schools, eighth-grade classrooms, and eighth-grade students in Colorado 
that participated in TIMSS, and their participation rates: 2011 



Number 

Rates 

Unweighted 

Weighted 

Schools 




Sampled 

60 



Excluded and ineligible 

0 



Eligible 

60 



Participating 

50 

83 

84 

Substitutes 

3 



Participating (all schools) 

53 

88 

89 

Classrooms in participating schools 




Total 

535 



Excluded 

21 



Eligible 

514 



Sampled 

97 



Participating 

97 

100 

100 

Students in participating schools 




Sampled 

2,395 



Excluded 

47 



Withdrawn 

60 



Eligible 

2,288 



Absent/refusal 

121 



Participating 

2,167 

95 

94 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the Colorado TIMSS grade 8 sample 

The Colorado TIMSS eighth grade sample had a (weighted) student exclusion rate of 4.1 percent based on 
the combination of whole -class and within-class exclusions. lEA standards define this degree of coverage 
of the national target population (95.9 percent) as acceptable. 
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Table B-4. Number of sampled schools, eighth-grade classrooms, and eighth-grade students in 
Connecticut that participated in TIMSS, and their participation rates: 2011 



Number 

Rates 

Unweighted 

Weighted 

Schools 




Sampled 

63 



Excluded and ineligible 

1 



Eligible 

62 

100 

100 

Participating 

62 



Substitutes 

0 



Participating (all schools) 

62 

100 

100 

Classrooms in participating schools 




Total 

680 



Excluded 

25 



Eligible 

655 



Sampled 

113 



Participating 

113 

100 

100 

Students in participating schools 




Sampled 

2,356 



Excluded 

115 



Withdrawn 

16 



Eligible 

2,225 



Absent/refusal 

126 



Participating 

2,099 

94 

94 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the Connecticut TIMSS grade 8 sample 

The Connecticut TIMSS eighth grade sample had a (weighted) student exclusion rate of 8.5 percent based 
on the combination of whole-class and within-class exclusions. lEA standards define this degree of 
coverage of the national target population (91.5 percent) as acceptable though falling below the desired 
range of 95 percent or better. The tabulations shown in the international reports show Connecticut 
annotated to indicate this fact. 
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Table B-5. Number of sampled schools, fourth-grade classrooms, and fourth-grade students in Florida 
that participated in PIRLS, and their participation rates: 2011 



Number 

Rates 

Unweighted 

Weighted 

Schools 




Sampled 

81 



Excluded and ineligible 

1 



Eligible 

80 

96 

96 

Participating 

77 



Substitutes 

0 



Participating (all schools) 

77 

96 

96 

Classrooms in participating schools 




Total 

456 



Excluded 

26 



Eligible 

430 



Sampled 

148 



Participating 

148 

99 

99 

Students in participating schools 




Sampled 

3,052 



Excluded 

269 



Withdrawn 

43 



Eligible 

2,740 



Absent/refusal 

142 



Participating 

2,598 

95 

95 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). Flowever, the participation rates shown in this 
table are the official international PIRLS rates, whose classroom and student participation rates do include 
substitute schools in their calculation. The exclusion rates for the PIRLS Florida fourth grade sample are 
presented following Table B-6. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Progress in 
International Reading Literacy Study (PIRLS), 2011. 
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Table B-6. Number of sampled schools, fourth- and eighth-grade classrooms, and students in Florida 
that participated in TIMSS, and their participation rates, by grade: 2011 



Number 


Rates 



Grade 4 

Grade 8 

Grade 4 

Unweighted Weighted 

Grade 8 

Unweighted Weighted 

Schools 







Sampled 

81 

65 





Excluded and ineligible 

1 

1 





Eligible 

80 

64 





Participating 

77 

60 

96 

96 

94 

94 

Substitutes 

0 

0 





Participating (all schools) 

77 

60 

96 

96 

94 

94 

Classrooms in participating schools 







Total 

456 

1,142 





Excluded 

26 

56 





Eligible 

430 

1,086 





Sampled 

149 

100 





Participating 

149 

100 

100 

100 

98 

98 

Students in participating schools 







Sampled 

3,121 

1,986 





Excluded 

265 

87 





Withdrawn 

43 

25 





Eligible 

2,813 

1,874 





Absent/refusal 

152 

162 





Participating 

2,661 

1,712 

95 

95 

91 

91 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of selection 


(National Center for Education Statistics, 2002). However, the participation rates shown in this table are the 
official international TIMSS rates, whose classroom and student participation rates do include substitute 
schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the Florida TIMSS and PIRLS grade 4 samples and TIMSS grade 8 sample 

Florida had a (weighted) student exclusion rate of 12.1 percent in the fourth-grade for TIMSS, 
12.9 percent for PIRLS, and 6.9 percent for TIMSS in the eighth-grade, based on the combination of 
whole-class and within-class exclusions. lEA standards define this degree of coverage of the national 
target population (87.9 percent for both TIMSS, 87.1 percent for PIRLS at fourth grade and 93.1 percent 
for eighth grade) as acceptable though falling below the desired range of 95 percent or better. The 
tabulations shown in the international reports show the Florida annotated to indicate this fact. 
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Table B-7. Number of sampled schools, eighth-grade classrooms, and eighth-grade students in Indiana 
that participated in TIMSS, and their participation rates: 2011 

Rates 



Number 

Unweighted 

Weighted 

Schools 




Sampled 

62 



Excluded and ineligible 

4 



Eligible 

58 



Participating 

55 

95 

94 

Substitutes 

1 



Participating (all schools) 

56 

97 

97 

Classrooms in participating schools 




Total 

652 



Excluded 

46 



Eligible 

606 



Sampled 

105 



Participating 

105 

100 

100 

Students in participating schools 




Sampled 

2,501 



Excluded 

97 



Withdrawn 

49 



Eligible 

2,355 



Absent/refusal 

95 



Participating 

2,260 

96 

96 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 

Exclusions in the Indiana TIMSS grade 8 sample 

The Indiana TIMSS eighth grade sample had a (weighted) student exclusion rate of 6.3 percent based on 
the combination of whole -class and within-class exclusions. lEA standards define this degree of coverage 
of the national target population (93.7 percent) as acceptable though falling below the desired range of 
95 percent or better. The tabulations shown in the international reports show Indiana annotated to indicate 
this fact. 
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Table B-8. Number of sampled schools, eighth-grade classrooms, and eighth-grade students in 
Massachusetts that participated in TIMSS, and their participation rates: 2011 



Number 

Rates 

Unweighted 

Weighted 

Schools 




Sampled 

58 



Excluded and ineligible 

2 



Eligible 

56 



Participating 

56 

100 

100 

Substitutes 

0 



Participating (all schools) 

56 

100 

100 

Classrooms in participating schools 




Total 

592 



Excluded 

47 



Eligible 

545 



Sampled 

103 



Participating 

103 

100 

100 

Students in participating schools 




Sampled 

2,296 



Excluded 

112 



Withdrawn 

20 



Eligible 

2,164 



Absent/refusal 

89 



Participating 

2,075 

96 

96 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the Massachusetts TIMSS grade 8 sample 

The Massachusetts TIMSS eighth grade sample had a (weighted) student exclusion rate of 7.9 percent 
based on the combination of whole-class and within-class exclusions. lEA standards define this degree of 
coverage of the national target population (92. 1 percent) as acceptable though falling below the desired 
range of 95 percent or better. The tabulations shown in the international reports show Massachusetts 
annotated to indicate this fact. 
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Table B-9. Number of sampled schools, eighth-grade classrooms, and eighth-grade students in 
Minnesota that participated in TIMSS, and their participation rates: 2011 



Numbers 

Rates 

Unweighted 

Weighted 

Schools 




Sampled 

60 



Excluded and ineligible 

4 



Eligible 

56 



Participating 

51 

91 

91 

Substitutes 

4 



Participating (all schools) 

55 

98 

98 

Classrooms in participating schools 




Total 

691 



Excluded 

65 



Eligible 

626 



Sampled 

102 



Participating 

102 

100 

100 

Students in participating schools 




Sampled 

2,720 



Excluded 

61 



Withdrawn 

32 



Eligible 

2,627 



Absent/refusal 

127 



Participating 

2,500 

95 

95 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the Minnesota TIMSS grade 8 sample 

The Minnesota TIMSS eighth grade sample had a (weighted) student exclusion rate of 4.3 percent based 
on the combination of whole-class and within-class exclusions. lEA standards define this degree of 
coverage of the national target population (95.7 percent) as acceptable. 
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Table B-10. Number of sampled schools, fourth- and eighth-grade classrooms, and students in North 
Carolina that participated in TIMSS, and their participation rates, by grade: 2011 



Numbers 

Rates 


Grade 4 

Grade 8 

Grade 4 Grade 8 

Unweighted Weighted Unweighted Weighted 

Schools 





Sampled 

49 

62 



Excluded and ineligible 

0 

2 



Eligible 

49 

60 



Participating 

46 

59 

94 94 98 

98 

Substitutes 

0 

0 



Participating (all schools) 

46 

59 

94 94 98 

98 

Classrooms in participating schools 





Total 

212 

626 



Excluded 

4 

48 



Eligible 

208 

578 



Sampled 

92 

103 



Participating 

92 

103 

100 100 100 

100 

Students in participating schools 





Sampled 

2,104 

2,434 



Excluded 

203 

203 



Withdrawn 

13 

24 



Eligible 

1,888 

2,207 



Absent/refusal 

96 

104 



Participating 

1,792 

2,103 

95 95 95 

95 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools because substitute schools do not have an independent probability of 
selection (National Center for Education Statistics, 2002). However, the participation rates shown in this 
table are the official international TIMSS rates, whose classroom and student participation rates do 
include substitute schools in their calculation. 

SOURCE: International Association for the Evaluation of Educational Achievement (lEA), Trends in 
International Mathematics and Science Study (TIMSS), 2011. 


Exclusions in the North Carolina TIMSS grades 4 and 8 samples 

North Carolina had a (weighted) student exclusion rate of 10. 1 percent in the fourth-grade for TIMSS and 
11.4 percent for TIMSS in the eighth-grade, based on the combination of whole -class and within-class 
exclusions. lEA standards define this degree of coverage of the national target population (89.9 percent 
for both TIMSS at fourth grade and 88.6 percent for eighth-grade) as acceptable though falling below the 
desired range of 95 percent or better. The tabulations shown in the international reports show North 
Carolina annotated to indicate this fact. 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


B-11 









APPENDIX C 


TIMSS & PIRLS 2011 Recruitment Materials 

C-1. TIMSS & PIRLS 2011 State Letter C-1 

C-2. TIMSS & PIRLS 2011 District Letter C-2 

C-3. TIMSS 2011 District Letter C-4 

C-4. TIMSS & PIRLS 2011 School Letter C-6 

C-5. TIMSS 2011 Eighth-Grade School Letter C-8 

C-6. TIMSS 2011 Fourth-Grade Notification Letter C-9 

C-7. TIMSS 2011 Fourth-Grade Implicit Consent Letter C-11 

C-8. TIMSS 2011 Fourth-Grade Explicit Consent Letter C-15 

C-9. TIMSS 2011 Fourth-Grade Facts for Parents C-19 

C-10. PIRLS 2011 Notification Letter C-22 

C-11. PIRLS 2011 Implicit Consent Letter C-24 

C-12. PIRLS 2011 Explicit Consent Letter C-28 

C-13. PIRLS 2011 Facts for Parents C-32 

C-14. TIMSS 2011 Eighth-Grade Notification Letter C-35 

C-15. TIMSS 2011 Eighth-Grade Facts for Parents C-37 

U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide C-1 



Exhibit C-1. TIMSS & PIRLS 2011 State Letter 

I 


State PIRLS & TIMSS letter 


[Date] 

[Title] [Name First] [Name Last] 

[T itle/Department] 

[State] 

[Address I] 

[Address 2] 

[City], [State] [Zip code] 

Dear [Title] [Name Last]: 

The United States will be participating in two important international studies in 20 1 1 to help benchmark student 
performance in the United States compared to that in other countries around the world. Some schools in your state 
have been randomly selected to participate in these studies in the spring of 201 1 . lam writing to ask your agency to 
support the participation of schools in your state in these two studies: the Trends in International Mathematics and 
Science Study (TIMSS) 20 1 1 and the Progress in International Reading Literacy Study (PIRLS) 201 1 . 

TIMSS is administered every four years in more than 60 countries and provides important information for 
internationally benchmarking U.S. performance in mathematics and science at the fourth- and eighth-grade levels 
against top countries around the world. PIRLS is administered every five years in more than 50 countries and 
provides similarly important international benchmarking information in fourth-grade reading. 

TIMSS, PIRLS, and the process for participating schools are described in more detail in materials enclosed with this 
letter. The studies are sponsored in the United States by the National Center for Education Statistics in the U.S. 
Department of Education and are conducted by Westat of Rockville, MD. The U.S. Office of Management and 
Budget has approved the data collection under 0MB # 1850-0645 v5. While participation in this study is entirely 
voluntary, we ask your agency to support participation on the part of schools in your state so that the United States 
has a representative sample of schools across the country. 

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Sectionl53). By law, the data provided by your schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107-279, Section 
183 and Title V, subtitle A of the E-Govemment Act of 2002 (P.L. 107-347)). Reports of the findings from the 
assessments will not identify participating districts, schools, students, or individual staff. Individual responses will 
be combined with those from other participants to produce summary statistics and reports. 

Within the next few weeks, a representative of Westat will contact sampled school districts and schools to discuss 
conducting data collection. In the meantime, if you have questions about the study, please do not hesitate to call 
[insert name of contact] at 1-888-369-5033 or send an email to TIMSS-PIRLS@westat.com . You may also get 
more information about these studies by contacting Dr. Patrick Gonzales at NCES at (415) 920-9229 or visiting the 
TIMSS and PIRLS websites at: http://nces.ed.gov/timss/ http://nces.ed.gov/survevs/pirls/ . 

Thank you for your time and support. TIMSS and PIRLS are important elements in the U.S. effort to benchmark the 
performance and progress of our education system against international standards. 

Sincerely, 


Stuart Kerachsky 

Acting Commissioner 

cc: [State assessment coordinator name] 

Enclosures 
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School district PIRLS & TIMSS letter 


[Date] 


[Title] [Name First] [Name Last], [Title/Department] 

[School District] 

[Address 1] 

[City], [State] [Zip code] 

Dear [Title] [Name Last]: 

The United States will be participating in two important international studies in 201 1 to help benchmark 
student performance in the United States against that in other countries around the world: the Trends in 
International Mathematics and Science Study (TIMSS) 2011 and the Progress in International Reading 
Literacy Study (PIRLS) 2011. We are contacting you because one or more schools in your diocese have 
been randomly selected to represent the United States in these studies in the spring of 20 1 1. 

TIMSS is administered every four years in more than 60 countries and provides important information for 
internationally benchmarking U.S. performance in mathematics and science at the fourth- and eighth- 
grade against top countries around the world. PIRLS is administered every five years in more than 50 
countries and provides similarly important international benchmarking information in fourth-grade 
reading. 


We ask your agency to support the participation of schools in your diocese in the TIMSS and PIRLS 
studies as these assessments are vital in understanding how the knowledge and skills of U.S. students 
compare with those of their peers in other countries. Schools that participate in the TIMSS and PIRLS 
studies will be partially compensated for their time and efforts: participating schools will receive $200, 
their school- level coordinator will receive $100, and each student that takes the assessment will receive a 
small gift. 

Materials enclosed with this letter describe TIMSS and PIRLS and the process for participating schools in 
more detail. Both assessments are sponsored by the U.S. Department of Education’s National Center for 
Education Statistics and are conducted by Westat, a research organization based in the Washington D.C. 
area. The U.S. Office of Management and Budget has approved this data collection under 0MB #1850- 
0645 v5. While participation in this study is entirely voluntary, we ask your agency to support the 
participation of schools in your diocese that the United States has a representative sample of schools 
across the country. 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 
107-279, Section 153). By law, the data provided by your schools, staff, and students may be used only 
for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose 
(Public Law 107-279, Section 183 and Title V, subtitle A of the E-Govemment Act of 2002 (P.L. 107- 
347)). We will disclose the names of schools in each district only to the governing district for each school, 
and we ask that each district maintain the confidentiality of the sampled schools in the TIMSS and PIRLS 
studies. Reports of the findings from these assessments will not identify participating districts, schools, 
students, or individual staff Individual responses will be combined with those from other participants to 
produce summary statistics and reports. 
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If you have any questions, please do not hesitate to eall the TIMSS and PIRLS Home Offiee at 1-888- 
369-5033 or send an email to TIMSS-PIRLS@westat.oom . You may also get more information about 
these studies by eontaeting Dr. Patriek Gonzales at NCES at (415) 920-9229 or visiting the TIMSS and 
PIRLS websites at: http ://noes .ed. gov/timss/ and http://nees.ed.gov/survevs/pirls/ . 

Thank you for your time and support. TIMSS and PIRLS are important elements in the U.S. effort to 
benehmark the performanee and progress of our edueation system against international standards. 

Sineerely, 


Stuart Kerachsky 
Aeting Commissioner 

Enelosures 
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School district TIMSS letter 


[Date] 


[Title] [Name First] [Name Last], [Title/Department] 
[School District] 

[Address 1] 

[City], [State] [Zip code] 


I am writing to inform you about the upcoming Trends in International Mathematics and Science Study 
(TIMSS) 2011, in which the United States will participate along with more than 60 other countries. 
TIMSS provides important international benchmarking information in fourth- and eighth-grade 
mathematics and science. We are notifying you now because one or more schools in your district have 
been randomly selected to take part in the TIMSS study in the spring of 201 1. 


We ask your agency to support the participation of schools in your district in the TIMSS assessment as it 
is vital to understanding how the knowledge and skills of U.S. students compare to those of their peers in 
other countries. Schools that participate in TIMSS will be compensated for their time and efforts. 
Participating schools will receive $200, their school-level coordinator will receive $100, and each student 
that takes the assessment will receive a small gift. 

Materials enclosed with this letter describe TIMSS and the process for participating schools in more 
detail. TIMSS is sponsored by the U.S. Department of Education’s National Center for Education 
Statistics and is conducted by Westat, a research organization based in the Washington D.C. area. The 
U.S. Office of Management and Budget has approved the data collection under 0MB #1850-0645 v5. 
While participation in this study is entirely voluntary, we ask your agency to support the participation of 
schools in your district so that the United States has a representative sample of schools across the country. 

Within the next few days, a representative of Westat will contact the following school or schools in your 
district that have been selected for the assessment: [LIST SAMPLED SCHOOLS HERE. . .] 

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 
107-279, Section 153). By law, the data provided by your schools, staff, and students may be used only 
for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose 
(Public Law 107-279, Section 183 and Title V, subtitle A of the E-Govemment Act of 2002 (P.L. 107- 
347)). We will disclose the names of schools in each district only to the governing district for each school, 
and we ask that each district maintain the confidentiality of the sampled schools in the TIMSS study. 
Reports of the findings from the assessment will not identify participating districts, schools, students, or 
individual staff. Individual responses will be combined with those from other participants to produce 
summary statistics and reports. 

If you have any questions, please do not hesitate to call the TIMSS Home Office at 1-888-369-5033 or 
send an email to TIMSS-PIRLS@westat.com . You may also get more information about these studies by 
contacting Dr. Patrick Gonzales at NCES at (415) 920-9229 or visiting the TIMSS website at: 
http://nces.ed.gov/timss/ . 

Thank you for your time and support. TIMSS is an important element in the U.S. effort to benchmark the 
performance and progress of our education system against international standards. 
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Sincerely, 


Stuart Kerachsky 
Acting Commissioner 

Enclosures 
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School PIRLS & TIMSS letter 


[Date] 


[Title] [Name First] [Name Last], [Title/Department] 

[Sehool Distriet] 

[Address 1] 

[City], [State] [Zip eode] 

Dear [Title] [Name Last]: 

The United States will be partieipating in two important international studies in 201 1 to help benehmark 
student performanee in the United States against that in other eountries around the world: the Trends in 
International Mathematics and Science Study (TIMSS) 2011 and the Progress in International Reading 
Literacy Study (PIRLS) 2011. We are contacting you because your school has been selected to represent 
the United States in the TIMSS and PIRLS international studies in the spring of 201 1. 

TIMSS is administered every four years in more than 60 countries and provides important information for 
internationally benchmarking U.S. performance in mathematics and science at the fourth- and eighth- 
grade levels against top countries around the world. PIRLS is administered every five years in more than 
50 countries and provides similarly important international benchmarking information in fourth-grade 
reading. 

I encourage your school’s participation in the TIMSS and PIRLS assessments as they are vital to 
understanding how the knowledge and skills of U.S. students compare with those of their peers in other 
countries. Schools that participate in the TIMSS and PIRLS studies will be compensated in part for their 
time and efforts: participating schools will receive $200, their school-level coordinator will receive $100, 
and each student that takes the assessment will receive a small gift. 

Materials enclosed with this letter describe TIMSS and PIRLS and the process for participating schools in 
more detail. Both assessments are sponsored in the United States by the U.S. Department of Education’s 
National Center for Education Statistics and are conducted by Westat of Rockville, MD. The U.S. Office 
of Management and Budget has approved the data collection under 0MB #1850-0645 v5. We hope you 
will participate in this study so that the United States has a representative sample of schools across the 
country; however, participation in this study is entirely voluntary. 

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 
107-279, Section 153). By law, the data provided by your schools, staff, and students may be used only 
for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose 
(Public Law 107-279, Section 183 and Title V, subtitle A of the E-Govemment Act of 2002 (P.L. 107- 
347)). We will only disclose the names of schools in each district to the governing district for each 
school, and we have asked that each district maintain the confidentiality of the sampled schools in the 
TIMSS and PIRLS study. Reports of the findings from these assessments will not identily participating 
districts, schools, students, or individual staff Individual responses will be combined with those from 
other participants to produce summary statistics and reports. 

Within the next few days, a representative of Westat will call you to discuss your participation in the 
assessment. In the meantime, if you have any questions about the TIMSS and PIRLS study or your 
school’s participation, please feel free to call 1-888-369-5033 or send an email to 
TIMSS-PIRLS@westat.com. You may also get more information about these studies by contacting Dr. 
Patrick Gonzales at NCES at (415) 920-9229 or visiting the TIMSS and PIRLS websites at: 
http://nces.ed.gov/timss/ and http://nces.ed.gov/survevs/pirls/ . 
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Thank you for your time and support. TIMSS and PIRLS are important elements in the U.S. effort to 
benchmark the performance and progress of our education system against international standards. 

Sincerely, 


Stuart Kerachsky 
Acting Commissioner 

Enclosures 
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School TIMSS letter 


[Date] 

[Title] [Name First] [Name Last], [Title/Department] 

[School District] 

[Address 1] 

[City], [State] [Zip code] 

Dear [Title] [Name Last]: 

I am writing to inform you about the upcoming Trends in International Mathematics and Science Study 
(TIMSS) 201 1, in which the United States will participate along with more than 60 other countries. TIMSS 
provides important international benchmarking information in fourth- and eighth-grade mathematics and 
science. We are notilying you now because your school has been randomly selected to take part in the TIMSS 
study in the spring of 201 1 . 


I encourage your school’s participation in the TIMSS assessment as it is vital in understanding how the 
knowledge and skills of U.S. students eompare with those of their peers in other eountries. Sehools that 
partieipate in TIMSS will be eompensated in part for their time and efforts: partieipating sehools will 
reeeive $200, their sehool-level eoordinator will reeeive $100, and eaeh student that takes the assessment 
will reeeive a small gift 

Materials enclosed with this letter describe TIMSS and the process for participating schools in more detail. 
Both assessments are sponsored in the United States by the U.S. Department of Education’s National Center 
for Education Statistics and are conducted by Westat of Rockville, MD. The U.S. Office of Management and 
Budget has approved the data collection under 0MB # 1850-0645 v5. We hope you will participate in this 
study so that the United States has a representative sample of schools across the country; however, 
participation in this study is entirely voluntary. 

NCES is authorized to conduct this study under the Education Sciences Refonn Act of 2002 (Public Law 107- 
279, Section 153). By law, the data provided by your schools, staff, and students may be used only for 
statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 
107-279, Section 183 and Title V, subtitle A of the E-Govemment Act of 2002 (P.L. 107-347)). We will only 
disclose the names of schools in each district to the governing district for each school, and we have asked that 
each district maintain the confidentiality of the sampled schools in the TIMSS study. Reports of the findings 
from the assessment will not identify participating districts, schools, students, or individual staff Individual 
responses will be combined with those from other participants to produce summary statistics and reports. 

Within the next few days, a representative of Westat will call you to discuss your participation in TIMSS. In 
the meantime, if you have any questions about the TIMSS or your school’s participation, please feel free to 
call 1-888-369-5033 or send an email to TIMSS-PIRLS@westat.com. You may also get more information 
about these studies by contacting Dr. Patrick Gonzales at NCES at (415) 920-9229 or visiting the TIMSS 
website at: http://nces.ed.gov/timss/ . 

Thank you for your time and support. TIMSS is an element in the U.S. effort to benchmark the performance 
and progress of our education system against international standards. 

Sincerely, 

Stuart Kerachsky 
Acting Commissioner 
Enclosures 
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Sample Notification Letter, TIMSS Grade 4 


SCHOOL LETTERHEAD 


Dear Parent or Guardian, 

Our school has accepted an invitation from the National Center for Education Statistics (NCES), 
U.S. Department of Education, to participate in an important international study of student learning. This 
study is called TIMSS for short; the full name is the Trends in International Mathematics and Science 
Study. TIMSS looks at student mathematics and science achievement in schools around the world and 
documents worldwide trends in student knowledge of mathematics and science since 1995. 

The enclosed summary sheet provides some background to TIMSS, explains what is involved for 
each student selected to participate in the study, and gives a contact phone number and email address 
where you can find answers to any questions you might have. 

{Insert number} of our 4th -grade classes will take part. {This/One of these} is your child's class. 
{This Class/These two classes}, along with some {500} other classes of 4th graders nationwide, will 
contribute to this picture of what U.S. 4th graders know about mathematics and science, and how they 
compare with 4th graders worldwide. 

To have an accurate picture of what U.S. 4th graders can do, it is important that each student 
selected take part in the study. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students 
actually enjoy taking part, and participating students will receive a small gift, which we think they will like. 

All of the information collected is kept completely confidential, as required by law. NCES is 
authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). By law, the data provided by schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107- 
279, Section 183 and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 

Thank you for taking the time to think about this. We wish you all the best. 

Sincerely, 


Enclosures: 

Facts About TIMSS for Parents 
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Exhibit C-6. TIMSS 2011 Fourth-Grade Notification Letter - Continued 


Sample Notification Letter, TIMSS Grade 4 


SCHOOL LETTERHEAD 


Estimado padre/madre o tutor, 

Nuestra escuela ha aceptado una invitacion del Centro Nacional de Estadfsticas de la Educaclon 
(NCES, por sus siglas en Ingles) del Departamento de Educaclon de Estados Unidos para participar en 
un Importante estudio Internaclonal sobre el aprendizaje estudlantll. El nombre corto de este estudio es 
TIMSS; abrevlaclon de Estudio Internaclonal Sobre las Tendenclas en Matematicas y Clenclas. Desde 
1995 TIMSS ha observado los logros de los estudlantes en matematicas y clenclas en las escuelas del 
mundo y ha documentado las tendenclas mundlales en el conocimlento estudlantll de matematicas y 
clenclas. 

El resumen adjunto ofrece Informaclon de trasfondo de TIMSS, explIca lo que Implica la 
participaclon en el estudio para cada estudlante selecclonado e Incluye un numero de telefono y un 
correo electronico de contacto donde usted podra encontrar respuestas a cualquier pregunta que tenga. 

{Insert number} de nuestras clases de cuarto grade participaran. {Esta/Una de estas} es la clase 
de su hijo. {Esta clase/Estas dos clases}, junto con otras {500} clases de estudlantes de cuarto grade en 
todo el pais, contrlbulran a mostrar lo que saben los estudlantes de cuarto grade en Estados Unidos 
sobre matematicas y clenclas, y como se comparan con estudlantes de cuarto grade alrededor del 
mundo. 


Para tener una Imagen precisa de lo que los estudlantes de cuarto grade en Estados Unidos 
pueden hacer, es Importante que cada estudlante selecclonado participe en el estudio. Lo Invito a que 
apoye este esfuerzo animando a su hIjo a participar; sin embargo, la participaclon en este estudio es 
completamente voluntarla. Las experlenclas anterlores parecen IndIcar que los estudlantes disfrutan de 
su participaclon. Ademas, los estudlantes que participen recibiran un pequeho regalo que creemos les 
gustara. 


Toda la Informaclon que se reuna es completamente confidenclal, como lo exige la ley. NCES 
esta autorizado a realizar este estudio de acuerdo con la Ley de reforma de clenclas de la educaclon de 
2002 (Ley publica 107-279, secclon 153). Por ley, la Informaclon dada por las escuelas, empleados y 
estudlantes solamente se puede utilizer con fines estadfsticos y no se puede dar a conocer nl utilizer de 
una manera que permita la Identificaclon de personas para otros fines (Ley publica 107-279, secclon 183 
y tftulo V, subtftulo A de la Ley de goblerno electronico de 2002 (Ley publica 107-347)). En los Informes 
nunca se Identifica ni a los estudlantes ni a las escuelas. Todas las estadfsticas publlcadas se refleren a 
Estados Unidos en conjunto. 

Gracias por tomarse el tiempo de considerar este estudio. Reciba nuestros mejores deseos. 
Atentamente, 


Se adjuntan los sigulentes documentos: 
Informaclon sobre TIMSS para los padres 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


C-11 




Exhibit C-7. TIMSS 2011 Fourth-Grade Impiicit Consent Letter 


Sample Implicit Consent Letter, TIMSS Grade 4 


SCHOOL LETTERHEAD 


Dear Parent or Guardian, 

Our school has accepted an invitation from the National Center for Education Statistics (NCES), 
U.S. Department of Education, to participate in an important international study of student learning. This 
study is called TIMSS for short; the full name is the Trends in International Mathematics and Science 
Study. TIMSS looks at student mathematics and science achievement in schools around the world and 
documents worldwide trends in student knowledge of mathematics and science since 1995. 

The enclosed summary sheet provides some background to TIMSS, explains what is involved for 
each student selected to participate in the study, and gives a contact phone number and email address 
where you can find answers to any questions you might have. 

{Insert number} of our 4th-grade classes will take part. {This/One of these} is your child's class. 
{This Class/These two classes}, along with some {500} other classes of 4th graders nationwide, will 
contribute to this picture of what U.S. 4th graders know about mathematics and science, and how they 
compare with 4th graders worldwide. 

To have an accurate picture of what U.S. 4th graders can do, it is important that each student 
selected take part in the study. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students 
actually enjoy taking part, and participating students will receive a small gift, which we think they will like. 

All of the information collected is kept completely confidential, as required by law. NCES is 
authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). By law, the data provided by schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107- 
279, Section 183 and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 

If you have any objection to your child joining in the TIMSS activities please let us know by 
completing the attached form (‘Being Part of TIMSS’) and returning it to the school. 

Thank you for taking the time to think about this. We wish you all the best. 

Sincerely, 


Enclosures: 

Facts About TIMSS for Parents 
Being Part of TIMSS 
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Exhibit C-7. TIMSS 2011 Fourth-Grade Impiicit Consent Letter - Continued 


TIMSS Implicit Consent Form 


Being Part of TIMSS 


We would like very much to have your child take part in the TIMSS project along with his/her classmates. 
Student participation in TIMSS is critical to the success of the study and, ultimately, to the nation’s interests 
in improving mathematics and science education. Your child will be included in TIMSS unless you indicate 
otherwise. If you do not want your child to participate, please check the “No” box below, sign the form and 
send it back to the school. The school will make arrangements for your child to undertake some other 
activities during the time that the other students are involved with TIMSS. 



No, I do not want my child to take part in the study. 


Child’s Name: 

First 


Middle 


Last 


Parent’s or Guardian’s Signature 


Date 


Thank you for taking the time to think about this very important project. 
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Exhibit C-7. TIMSS 2011 Fourth-Grade Impiicit Consent Letter - Continued 


Sample Implicit Consent Letter, TIMSS Grade 4 


SCHOOL LETTERHEAD 


Estimado padre/madre o tutor, 

Nuestra escuela ha aceptado una invitacion del Centro Nacional de Estadfsticas de la Educaclon 
(NCES, por sus siglas en Ingles) del Departamento de Educaclon de Estados Unidos para participar en 
un Importante estudio Internaclonal sobre el aprendizaje estudlantll. El nombre corto de este estudio es 
TIMSS; abrevlaclon de Estudio Internaclonal Sobre las Tendenclas en Matematicas y Clenclas. Desde 
1995 TIMSS ha observado los logros de los estudlantes en matematicas y clenclas en las escuelas del 
mundo y ha documentado las tendenclas mundlales en el conocimlento estudlantll de matematicas y 
clenclas. 

El resumen adjunto ofrece Informaclon de trasfondo de TIMSS, explIca lo que Implica la 
participaclon en el estudio para cada estudlante selecclonado e Incluye un numero de telefono y un 
correo electronico de contacto donde usted podra encontrar respuestas a cualquier pregunta que tenga. 

{Insert number} de nuestras clases de cuarto grade participaran. {Esta/Una de estas} es la clase 
de su hijo. {Esta clase/Estas dos clases}, junto con otras {500} clases de estudlantes de cuarto grade en 
todo el pais, contrlbulran a mostrar lo que saben los estudlantes de cuarto grade en Estados Unidos 
sobre matematicas y clenclas, y como se comparan con estudlantes de cuarto grade alrededor del 
mundo. 


Para tener una Imagen precisa de lo que los estudlantes de cuarto grade en Estados Unidos 
pueden hacer, es Importante que cada estudlante selecclonado participe en el estudio. Lo Invito a que 
apoye este esfuerzo animando a su hIjo a participar; sin embargo, la participaclon en este estudio es 
completamente voluntarla. Las experlenclas anterlores parecen IndIcar que los estudlantes disfrutan de 
su participaclon. Ademas, los estudlantes que participen recibiran un pequeho regalo que creemos les 
gustara. 


Toda la Informaclon que se reuna es completamente confidenclal, como lo exige la ley. NCES 
esta autorizado a realizar este estudio de acuerdo con la Ley de reforma de clenclas de la educaclon de 
2002 (Ley publica 107-279, secclon 153). Por ley, la Informaclon dada por las escuelas, empleados y 
estudlantes solamente se puede utilizer con fines estadfsticos y no se puede dar a conocer ni utilizer de 
una manera que permita la Identificaclon de personas para otros fines (Ley publica 107-279, secclon 183 
y tftulo V, subtftulo A de la Ley de goblerno electronico de 2002 (Ley publica 107-347)). En los Informes 
nunca se Identifica ni a los estudlantes ni a las escuelas. Todas las estadfsticas publicadas se refieren a 
Estados Unidos en conjunto. 

Si tiene alguna objecion en que su hijo participe en las actividades de TIMSS, por favor 
haganoslo saber al completar el formulario adjunto ('Participaclon en TIMSS') y enviarlo a la escuela lo 
antes posible. 

Gracias por tomarse el tiempo de considerar este estudio. Reciba nuestros mejores deseos. 
Atentamente, 


Se adjuntan los siguientes documentos: 

Informaclon sobre TIMSS para los padres 
Participaclon en TIMSS 
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Exhibit C-7. TIMSS 2011 Fourth-Grade Impiicit Consent Letter - Continued 


TIMSS Implicit Consent Form 


Participacion en TIMSS 


Nos encantaria que su hijo participara en el proyecto TIMSS junto con sus companeros de clase. La 
participacion de los estudiantes en TIMSS es sumamente importante para el exito del estudio y en ultima 
instancia para los intereses del pais de mejorar la educacion en matematicas y ciencias. Su hijo sera incluido 
en TIMSS a menos que usted indique lo contrario. Si no desea que su hijo participe, por favor marque la 
casilla "No" a continuacion, firme el formulario y envielo a la escuela. La escuela se asegurara que su hijo 
realice otra actividad en el tiempo en que los demas estudiantes esten participando en TIMSS. 


No, no deseo que mi hijo participe en el estudio. 


Nombre del nino: 


Nombre complete 


Apellido 


Firma del padre o madre o guardian 


Fecha 


Gracias por tomarse el tiempo de considerar este importante proyecto. 
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Exhibit C-8. TIMSS 2011 Fourth-Grade Expiicit Consent Letter 

Sample Explicit Consent Letter, TIMSS Grade 4 


SCHOOL LETTERHEAD 


Dear Parent or Guardian, 

Our school has accepted an invitation from the National Center for Education Statistics (NCES), 
U.S. Department of Education, to participate in an important international study of student learning. This 
study is called TIMSS for short; the full name is the Trends in International Mathematics and Science 
Study. TIMSS looks at student mathematics and science achievement in schools around the world and 
documents worldwide trends in student knowledge of mathematics and science since 1995. 

The enclosed summary sheet provides some background to TIMSS, explains what is involved for 
each student selected to participate in the study, and gives a contact phone number and email address 
where you can find answers to any questions you might have. 

{Insert number} of our 4th-grade classes will take part. {This/One of these} is your child's class. 
{This Class/These two classes}, along with some {500} other classes of 4th graders nationwide, will 
contribute to this picture of what U.S. 4th graders know about mathematics and science, and how they 
compare with 4th graders worldwide. 

To have an accurate picture of what U.S. 4th graders can do, it is important that each student 
selected take part in the study. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students 
actually enjoy taking part. And participating students will receive a small gift, which we think they will like. 

All of the information collected is kept completely confidential, as required by law. NCES is 
authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). By law, the data provided by schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107- 
279, Section 183 and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 

Before we can allow your child to join in the TIMSS activities we must have your written consent. 
You can let us know by completing the attached form (‘Being Part of TIMSS’) and returning it to the 
school by {Insert date at least 2 days prior to assessment date}. 

Thank you for taking the time to think about this. We wish you all the best. 


Sincerely, 


Enclosures: 

Facts About TIMSS for Parents 
Being Part of TIMSS 
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Exhibit C-8. TIMSS 2011 Fourth-Grade Expiicit Consent Letter - Continued 

TIMSS Explicit Consent Form 


Being Part of TIMSS 


We need your consent. 

We would like very much to have your child take part in the TIMSS project along with his/her classmates. 
Student participation in TIMSS is critical to the success of the study and, ultimately, to the nation’s interests 
in improving mathematics and science education. Please check the “Yes” box below if you give your 
consent, or check the “No” box if you do not consent. If you do not want your child to participate, the 
school will make arrangements for your child to undertake some other activities during the time that the 
other students are involved with TIMSS. After making your selection, please sign the form and send it back 
to the school by {Insert date at least 2 days prior to assessment date} . 



Yes, my child may take part in TIMSS. 



No, I do not want my child to take part in the study. 


Child’s Name: 


First 


Middle 


Last 


Parent’s or Guardian’s Signature 


Date 


Thank you for taking the time to think about this very important project 
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Exhibit C-8. TIMSS 2011 Fourth-Grade Expiicit Consent Letter - Continued 

Sample Explicit Consent Letter, TIMSS Grade 4 


SCHOOL LETTERHEAD 


Estimado padre/madre o tutor, 

Nuestra escuela ha aceptado una invitacion del Centro Nacional de Estadisticas de la Educaclon 
(NCES, por sus siglas en Ingles) del Departamento de Educaclon de Estados Unldos para participar en 
un Importante estudio Internaclonal sobre el aprendizaje estudlantll. El nombre corto de este estudio es 
TIMSS; abrevlaclon de Estudio Internaclonal Sobre las Tendenclas en Matematicas y Clenclas. Desde 
1995 TIMSS ha observado los logros de los estudlantes en matematicas y clenclas en las escuelas del 
mundo y ha documentado las tendenclas mundlales en el conocimlento estudlantll de matematicas y 
clenclas. 

El resumen adjunto ofrece Informaclon de trasfondo de TIMSS, explIca lo que Implica la 
participaclon en el estudio para cada estudlante selecclonado e Incluye un numero de telefono y un 
correo electronico de contacto donde usted podra encontrar respuestas a cualquier pregunta que tenga. 

{Insert number} de nuestras clases de cuarto grade participaran. {Esta/Una de estas) es la clase 
de su hijo. {Esta clase/Estas dos clases}, junto con otras {500} clases de estudlantes de cuarto grade en 
todo el pais, contrlbulran a mostrar lo que saben los estudlantes de cuarto grade en Estados Unldos 
sobre matematicas y clenclas, y come se comparan con estudlantes de cuarto grade alrededor del 
mundo. 

Para tener una Imagen precisa de lo que los estudlantes de cuarto grade en Estados Unldos 
pueden hacer, es Importante que cada estudlante selecclonado participe en el estudio. Lo Invito a que 
apoye este esfuerzo animando a su hIjo a participar; sin embargo, la participaclon en este estudio es 
completamente voluntarla. Las experlenclas anterlores parecen IndIcar que los estudlantes disfrutan de 
su participaclon. Ademas, los estudlantes que participen recibiran un pequeho regalo que creemos les 
gustara. 


Toda la Informaclon que se reuna es completamente confidenclal, como lo exige la ley. NCES 
esta autorizado a realizar este estudio de acuerdo con la Ley de reforma de clenclas de la educaclon de 
2002 (Ley publica 107-279, secclon 153). Por ley, la Informaclon dada por las escuelas, empleados y 
estudlantes solamente se puede utilizer con fines estadisticos y no se puede dar a conocer nl utilizer de 
una manera que permita la Identificaclon de personas para otros fines (Ley publica 107-279, secclon 183 
y titulo V, subtitulo A de la Ley de goblerno electronico de 2002 (Ley publica 107-347)). En los Informes 
nunca se Identifica nl a los estudlantes nl a las escuelas. Todas las estadisticas publlcadas se refleren a 
Estados Unldos en conjunto. 

Antes de que podamos permitirle al nlho participar en las actividades de TIMSS, debemos tener 
su autorizaclon por escrito. Puede dejarnoslo saber completando el formularlo que se adjunta 
('Participaclon en TIMSS') y enviandolo a la escuela antes de {Insert date at least 2 days prior to 
assessment date}. 

Gracias por tomarse el tiempo de considerar este estudio. Reciba nuestros mejores deseos. 


Atentamente, 


Se adjuntan los sigulentes documentos: 

Informaclon sobre TIMSS para los padres 
Participaclon en TIMSS 
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Exhibit C-8. TIMSS 2011 Fourth-Grade Expiicit Consent Letter - Continued 

TIMSS Explicit Consent Form 


Participacion en TIMSS 


Necesitamos su autorizacion. 

Nos encantaria que su hijo participara en el proyecto TIMSS junto con sus companeros de clase. La 
participacion de los estudiantes en TIMSS es sumamente importante para el exito del estudio y en ultima 
instancia para los intereses del pais de mejorar la educacion en matematicas y ciencias. For favor marque 
"Si" a continuacion si da su autorizacion, o marque "No" si no la da. Si usted no desea que su hijo participe, 
la escuela se asegurara que su hijo realice otra actividad en el tiempo en que los demas estudiantes esten 
participando en TIMSS. Despues de marcar su eleccion, por favor firme el formulario y envielo a la escuela 
antes de {Insert date at least 2 days prior to assessment date} . 



Si, mi hijo puede participar en TIMSS. 



No, no deseo que mi hijo participe en el estudio. 


Nomhre del niho: 


Nombre complete 


Apellido 


Gracias por tomarse el tiempo de considerar este importante proyecto. 
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Exhibit C-9. TIMSS 2011 Fourth-Grade Facts for Parents 

Main study’s ‘Facts About TIMSS for Parents’ - Grade 4 

Facts About TIMSS for Parents 

In April and May of this year, your child’s school will join schools across the U.S. and around the world 
taking part in TIMSS, the Trends in International Mathematics and Science Study. The schools were 
selected randomly to represent the nation’s schools, and within each school, 4th-grade students were 
selected randomly to represent the nation’s 4th graders. Your child is among the 4th graders selected from 
this school to take part in TIMSS. 

What is TIMSS? 

TIMSS is an international assessment that measures student learning in mathematics and science. 
Periodically (199S, 2003, 2007, and now 2011) TIMSS documents worldwide trends in the knowledge of 
4th graders. The National Center for Education Statistics within the U.S. Department of Education 
sponsors U.S. participation in TIMSS. Along with more than 60 other nations, we will take part in the 201 1 
cycle just as we did in 199S, 2003, and 2007. Participation in this study is voluntary. 

What is involved? 

Prom April through May 201 1 , TIMSS staff will visit the school and administer an assessment that contains 
mathematics and science items. The assessment runs for 70 minutes with breaks between sections. Students 
will also receive a background questionnaire, which takes 20 to 30 minutes to complete. 

What are the benefits? 

The nation as a whole benefits from the contribution your child’s school makes to the national picture of 
what our 4th graders know about mathematics and science, and how they compare with 4th graders 
worldwide Schools benefit too since we provide each school with a report about how it did in the 
assessment. Last, and certainly not least, students receive a small gift that we believe they will like. 

Who administers TIMSS? 

The entire assessment is administered by trained staff from Westat, a research organization under contract 
to the U.S. Department of Education’s National Center for Education Statistics. 

All of the information collected is kept completely confidential, as required by law. NCES is authorized to 
conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, Section 153). 
By law, the data provided by schools, staff, and students may be used only for statistical purposes and may 
not be disclosed, or used, in identifiable form for any other purpose (Public Law 107-279, Section 183 and 
Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and schools are never 
identified in any reports. All reported statistics refer to the United States as a whole. 

Where can I find out more about TIMSS? 

There is a lot of information available through the TIMSS website at http: / / nces.ed.gov/ timss/ or 
http:/ /timss. bc.edu/ . Or, if you would like to contact a TIMSS staff member directly, please feel free to 
call the TIMSS hotline at 1 (888) 369-5033 or email us at TIMSS-PIRLS@westat.com. 
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Exhibit C-9. TIMSS 2011 Fourth-Grade Facts for Parents - Continued 

Main study’s ‘Facts About TIMSS for Parents’ - Grade 4 

Informacion sobre TIMSS para los 
padres 

En abril y mayo de este ano, la escuela de su nino participara junto con otras escuelas de Estados Unidos y 
de todo el mundo en el Estudio Internacional Sobre las Tendencias en Matematicas y Ciencias, TIMSS, por 
sus siglas en ingles. Las escuelas fueron seleccionadas al azar para representar a las escuelas de nuestro pals, 
y dentro de cada escuela, se seleccionaron al azar a estudiantes de cuarto grado para representar a los 
estudiantes de cuarto grado del pals. Su nino es uno de los estudiantes de cuarto grado seleccionados de esta 
escuela para participar en TIMSS. 

iQue es TIMSS? 

TIMSS es una evaluacion internacional que mide el aprendizaje de los estudiantes en matematicas y ciencias. 
TIMSS documenta periodicamente (1995, 2003, 2007 y ahora en 2011) las tendencias mundiales en el 
conocimiento de los estudiantes de cuarto grado. El Centro Nacional de Estadisticas de la Educacion dentro 
del Departamento de Educacion de Estados Unidos patrocina la participacion de Estados Unidos en TIMSS. 
Junto con mas de 60 pai'ses, participaremos en el ciclo del 20 1 1 tal como lo hicimos en 1 995, 2003 y en el 
2007. La participacion en este estudio es voluntaria. 

^Que implica? 

Desde abril hasta mayo de 2011, el personal de TIMSS visitara la escuela y administrara una 
evaluacion que contiene temas de lectura. La evaluacion dura 70 minutos y tiene descansos entre las 
secciones. A los estudiantes tambien se les dara un cuestionario sobre informacion general, que toma 
de 20 a 30 minutos para responder. 

^Cuales son los beneficios? 

El pais en conjunto se beneficia de la contribucion que la escuela del nino hace para crear una imagen del 
pais sobre lo que los estudiantes de cuarto grado saben de matematicas y ciencias y como se comparan con 
los estudiantes de cuarto grado a nivel mundial. Las escuelas tambien se benefician ya que a cada una le 
damos un informe sobre cual fue su desempeno en la prueba. Einalmente, los estudiantes reciben un 
pequeno regain que creemos les gustara. 

^Quien administra TIMSS? 

La totalidad de la evaluacion la administra personal capacitado de Westat, una compania de estudios de 
investigacion que tiene un contrato con el Centro Nacional de Estadisticas de la Educacion del 
Departamento de Educacion de Estados Unidos. 

Toda la informacion que se reuna sera completamente confidencial, como lo exige la ley. NCES esta 
autorizado a realizar este estudio de acuerdo con la Ley de reforma de ciencias de la educacion de 2002 (Ley 
publica 107-279, seccion 153). Bajo esa ley, la informacion dada por las escuelas, empleados y estudiantes 
solamente se puede utilizar con fines estadisticos y no se puede dar a conocer ni utilizar de una manera que 
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Exhibit C-9. TIMSS 2011 Fourth-Grade Facts for Parents - Continued 


permita la identificacion de personas para otros fines (Ley piiblica 107-279, seccion 183 y ti'tulo V, subti'tulo 
A de la Ley de gobierno electronico de 2002 (Ley piiblica 107-347)). En los informes nunca se identifica ni 
a los estudiantes ni a las escuelas. Todas las estadisticas publicadas se refieren a Estados Unidos en conjunto. 

^Donde puedo obtener mas informacion sobre TIMSS? 

Hay bastante informacion disponible en la pagina en Internet de TIMSS en http: / / nces.ed.gov/ timss/ o 
http:/ /timss. bc.edu/ . Si prefiere comunicarse con un empleado de TIMSS directamente, no dude en 
llamar a la linea de informacion de TIMSS al 1 (888) 369-S033 o enviarnos un correo electronico a TIMSS- 
PIRLS@westat . com . 
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Exhibit C-10. PIRLS 2011 Notification Letter 

Sample Notification Letter, PIRLS Grade 4 


SCHOOL LETTERHEAD 


Dear Parent or Guardian, 

Our school has accepted an invitation from the National Center for Education Statistics (NCES), 
U.S. Department of Education, to participate in an important international study of student learning. This 
study is called PIRLS for short; the full name is the Progress in International Reading Literacy Study. 
PIRLS looks at student reading achievement in schools around the world and, every five years since 
2001, documents worldwide trends in reading literacy. 

The enclosed summary sheet provides some background to PIRLS, explains what is involved for 
each student selected to participate in the study, and gives a contact phone number and email address 
where you can find answers to any questions you might have. 

{Insert number} of our 4th-grade classes will take part in PIRLS. {This/One of these} is your child's 
class. {This Class/These classes}, along with some {500} other classes of 4th graders nationwide, will 
contribute to this picture of the reading achievement of U.S. 4th graders, and how they compare with 4th 
graders worldwide. 

To have an accurate picture of what U.S. 4th graders can do, it is important that each student 
selected take part in the study. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students 
actually enjoy taking part, and participating students will receive a small gift, which we think they will like. 

All of the information collected is kept completely confidential, as required by law. NCES is 
authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). By law, the data provided by schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107- 
279, Section 183 and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 

Thank you for taking the time to think about this. We wish you all the best. 


Sincerely, 


Enclosures: 
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Exhibit C-10. PIRLS 2011 Notification Letter - Continued 

Sample Notification Letter, PIRLS Grade 4 


SCHOOL LETTERHEAD 


Estimado padre/madre o tutor, 

Nuestra escuela ha aceptado una invitacion del Centro Nacional de Estadfsticas de la Educaclon 
(NCES, por sus siglas en Ingles) del Departamento de Educaclon de Estados Unidos para participar en 
un Importante estudio Internaclonal sobre el aprendizaje estudlantll. El nombre corto de este estudio es 
PIRLS; abrevlaclon de Estudio Internaclonal Sobre el Progreso en el Aprendizaje de la Lectura. Desde 
2001 PIRLS ha observado los logros de los estudlantes en lectura en las escuelas del mundo y cada 
cinco ahos documenta las tendenclas mundlales en el aprendizaje de la lectura. 

El resumen adjunto ofrece Informaclon de trasfondo de PIRLS, explIca lo que Implica la 
participaclon en el estudio para cada estudlante selecclonado e Incluye un numero de telefono y un 
correo electronico de contacto donde usted podra encontrar respuestas a cualquier pregunta que tenga. 

{Insert number} de nuestras clases de cuarto grade participaran. {Esta/Una de estas} es la clase 
de su hijo. {Esta clase/Estas dos clases}, junto con otras {500} clases de estudlantes de cuarto grade en 
todo el pais, contrlbulran a mostrar los logros en lectura de los estudlantes de cuarto grade en Estados 
Unidos, y como se comparan con estudlantes de cuarto grade alrededor del mundo. 

Para tener una Imagen precisa de lo que los estudlantes de cuarto grade en Estados Unidos 
pueden hacer, es Importante que cada estudlante selecclonado participe en el estudio. Lo Invito a que 
apoye este esfuerzo animando a su hijo a participar; sin embargo, la participaclon en este estudio es 
completamente voluntaria. Las experiencias anteriores parecen indicar que los estudlantes disfrutan de 
su participaclon. Ademas, los estudlantes que participen recibiran un pequeho regalo que creemos les 
gustara. 


Toda la informaclon que se reuna es completamente confidencial, como lo exige la ley. NCES 
esta autorizado a realizar este estudio de acuerdo con la Ley de reforma de ciencias de la educaclon de 
2002 (Ley publica 107-279, seccion 153). Por ley, la informaclon dada por las escuelas, empleados y 
estudlantes solamente se puede utilizer con fines estadfsticos y no se puede dar a conocer ni utilizar de 
una manera que permita la identificacion de personas para otros fines (Ley publica 107-279, seccion 183 
y tftulo V, subtftulo A de la Ley de gobierno electronico de 2002 (Ley publica 107-347)). En los informes 
nunca se identifica ni a los estudlantes ni a las escuelas. Todas las estadfsticas publicadas se refieren a 
Estados Unidos en conjunto. 

Gracias por tomarse el tiempo de considerar este estudio. Reciba nuestros mejores deseos. 
Atentamente, 


Se adjuntan los siguientes documentos: 
Informaclon sobre PIRLS para los padres 
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Exhibit C-11. PIRLS 2011 Impiicit Consent Letter 


Sample Implicit Consent Letter, PIRLS Grade 4 


SCHOOL LETTERHEAD 


Dear Parent or Guardian, 

Our school has accepted an invitation from the National Center for Education Statistics (NCES), 
U.S. Department of Education, to participate in an important international study of student learning. This 
study is called PIRLS for short; the full name is the Progress in International Reading Literacy Study. 
PILRS looks at student reading achievement in schools around the world and, every five years since 
2001, documents worldwide trends in reading literacy. 

The enclosed summary sheet provides some background to PIRLS, explains what is involved for 
each student selected to participate in the study, and gives a contact phone number and email address 
where you can find answers to any questions you might have. 

{Insert number} of our 4th-grade classes will take part. {This/One of these} is your child's class. 
{This Class/These classes}, along with some {500} other classes of 4th graders nationwide, will contribute 
to this picture of the reading achievement of U.S. 4th graders, and how they compare with 4th graders 
worldwide. 

To have an accurate picture of what U.S. 4th graders can do, it is important that each student 
selected take part in the study. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students 
actually enjoy taking part, and participating students will receive a small gift, which we think they will like. 

All of the information collected is kept completely confidential, as required by law. NCES is 
authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). By law, the data provided by schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107- 
279, Section 183 and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 

If you have any objection to your child joining in the PIRLS activities please let us know by 
completing the attached form (‘Being Part of PIRLS’) and returning it to the school. 

Thank you for taking the time to think about this. We wish you all the best. 


Enclosures: 

Facts About PIRLS for Parents 
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Exhibit C-11. PIRLS 2011 Impiicit Consent Letter - Continued 


PIRLS Implicit Consent Form 


Being Part of PIRLS 


We would like very much to have your child take part in the PIRLS project along with his/her classmates. 
Student participation in PIRLS is critical to the success of the study and, ultimately, to the nation’s interests 
in improving reading education. Your child will be included in PIRLS unless you indicate otherwise. 


If you do not want your child to participate in PIRLS, please check the “No” box below, sign the form, and 
send it back to the school. The school will make arrangements for your child to undertake some other 
activities during the time that the other students are involved with PIRLS. 



No, I do not want my child to take part in the study. 


Child’s Name: 

First 


Middle 


Last 


Parent’s or Guardian’s Signature 


Date 


Thank you for taking the time to think about this very important project. 
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Exhibit C-11. PIRLS 2011 Impiicit Consent Letter - Continued 


Sample Implicit Consent Letter, PIRLS Grade 4 


SCHOOL LETTERHEAD 


Estimado padre/madre o tutor, 

Nuestra escuela ha aceptado una invitacion del Centro Nacional de Estadfsticas de la Educaclon 
(NCES, por sus siglas en Ingles) del Departamento de Educaclon de Estados Unidos para participar en 
un Importante estudio Internaclonal sobre el aprendizaje estudlantll. El nombre corto de este estudio es 
PIRLS; abrevlaclon de Estudio Internaclonal Sobre el Progreso en el Aprendizaje de la Lectura. Desde 
2001 PIRLS ha observado los logros de los estudlantes en lectura en las escuelas del mundo y cada 
cinco ahos documenta las tendenclas mundlales en el aprendizaje de la lectura. 

El resumen adjunto ofrece Informaclon de trasfondo de PIRLS, explIca lo que Implica la 
participaclon en el estudio para cada estudlante selecclonado e Incluye un numero de telefono y un 
correo electronico de contacto donde usted podra encontrar respuestas a cualquier pregunta que tenga. 

{Insert number} de nuestras clases de cuarto grade participaran. {Esta/Una de estas} es la clase 
de su hijo. {Esta clase/Estas dos clases}, junto con otras {500} clases de estudlantes de cuarto grade en 
todo el pais, contrlbulran a mostrar los logros en lectura de los estudlantes de cuarto grade en Estados 
Unidos, y como se comparan con estudlantes de cuarto grade alrededor del mundo. 

Para tener una Imagen precisa de lo que los estudlantes de cuarto grade en Estados Unidos 
pueden hacer, es Importante que cada estudlante selecclonado participe en el estudio. Lo Invito a que 
apoye este esfuerzo animando a su hIjo a participar; sin embargo, la participaclon en este estudio es 
completamente voluntarla. Las experlenclas anterlores parecen IndIcar que los estudlantes disfrutan de 
su participaclon. Ademas, los estudlantes que participen recibiran un pequeho regalo que creemos les 
gustara. 


Toda la Informaclon que se reuna es completamente confidenclal, como lo exige la ley. NCES 
esta autorizado a realizar este estudio de acuerdo con la Ley de reforma de clenclas de la educaclon de 
2002 (Ley pubilea 107-279, secclon 153). Por ley, la Informaclon dada por las escuelas, empleados y 
estudlantes solamente se puede utilizer con fines estadfsticos y no se puede dar a conocer ni utilizer de 
una manera que permits la Identificaclon de personas para otros fines (Ley pubIlea 107-279, secclon 183 
y tftulo V, subtftulo A de la Ley de goblerno electronico de 2002 (Ley publics 107-347)). En los Informes 
nunca se Identifies ni a los estudlantes ni a las escuelas. Todas las estadfsticas publicadas se refieren a 
Estados Unidos en conjunto. 

Si tiene alguna objecion en que su hijo participe en las actividades de PIRLS, por favor 
haganoslo saber al completar el formulario adjunto (‘Participaclon en PIRLS’) y enviarlo a la escuela lo 
antes posible. 

Gracias por tomarse el tiempo de considerar este estudio. Reciba nuestros mejores deseos. 
Atentamente, 


Se adjuntan los siguientes documentos: 

Informaclon sobre PIRLS para los padres 
Participaclon en PIRLS 
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Exhibit C-11. PIRLS 2011 Impiicit Consent Letter - Continued 


PIRLS Implicit Consent Form 


Participacion en PIRLS 


Nos encantaria que su hijo participara en el proyecto PIRLS junto con sus companeros de clase. La 
participacion de los estudiantes en PIRLS es sumamente importante para el exito del estudio y en ultima 
instancia para los intereses del pais de mejorar la educacion en lectura. Su hijo sera incluido en PIRLS a 
menos que usted indique lo contrario. 

Si no desea que su hijo participe, por favor marque la casilla "No" a continuacion, firme el formulario y 
envielo a la escuela. La escuela se asegurara que su hijo realice otra actividad en el tiempo en que los demas 
estudiantes esten parti cipando en PIRLS. 



No, no deseo que mi hijo participe en el estudio. 


Nombre del nino: 


Nombre complete 


Apellido 


Firma del padre o madre o guardian 


Fecha 


Gracias por tomarse el tiempo de considerar este importante proyecto. 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


C-28 



Exhibit C-12. PIRLS 2011 Expiicit Consent Letter 


Sample Explicit Consent Letter, PIRLS Grade 4 


SCHOOL LETTERHEAD 


Dear Parent or Guardian, 

Our school has accepted an invitation from the National Center for Education Statistics (NCES), 
U.S. Department of Education, to participate in an important international study of student learning. This 
study is called PIRLS for short; the full name is the Progress in International Reading Literacy Study. 
PIRLS looks at student reading achievement in schools around the world and, every five years since 
2001, documents worldwide trends in reading literacy. 

The enclosed summary sheet provides some background to PIRLS, explains what is involved for 
each student selected to participate in the study, and gives a contact phone number and email address 
where you can find answers to any questions you might have. 

{Insert number} of our 4th-grade classes will take part. {This/One of these} is your child's class. 
{This Class/These two classes}, along with some {500} other classes of 4th graders nationwide, will 
contribute to this picture of the reading achievement of U.S. 4th graders, and how they compare with 4th 
graders worldwide. 

To have an accurate picture of what U.S. 4th graders can do, it is important that each student 
selected take part in the study. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students 
actually enjoy taking part, and participating students will receive a small gift, which we think they will like. 

All of the information collected is kept completely confidential, as required by law. NCES is 
authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). By law, the data provided by schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107- 
279, Section 183 and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 

Before we can allow your child to join in the PIRLS activities we must have your written consent. 
You can let us know by completing the attached form (‘Being Part of PIRLS’) and returning it to the 
school. 

Thank you for taking the time to think about this. We wish you all the best. 

Sincerely, 


Enclosures: 

Facts About PIRLS for Parents 
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Exhibit C-12. PIRLS 2011 Expiicit Consent Letter - Continued 


PIRLS Explicit Consent Form 

Being Part of PIRLS 


We need your consent. 


We would like very much to have your child take part in the PIRLS project along with his/her classmates. 
Student participation in PIRLS is critical to the success of the study and, ultimately, to the nation’s interests 
in improving reading education. Please check the “Yes” box below if you give your consent, or check the 
“No” box if you do not consent. If you do not want your child to participate, the school will make 
arrangements for your child to undertake some other activities during the time that the other students are 
involved with PIRLS. After making your selection, please sign the form and send it back to the school by 
{Insert date at least 2 days prior to assessment date} . 



Yes, my child may take part in PIRLS. 



No, I do not want my child to take part in the study. 


Child’s Name: 


First 


Middle 


Last 


Parent’s or Guardian’s Signature 


Date 


Thank you for taking the time to think about this very important project. 
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Exhibit C-12. PIRLS 2011 Expiicit Consent Letter - Continued 


Sample Explicit Consent Letter, PIRLS Grade 4 


SCHOOL LETTERHEAD 


Estimado padre/madre o tutor, 

Nuestra escuela ha aceptado una invitacion del Centro Nacional de Estadfsticas de la Educaclon 
(NCES, por sus siglas en Ingles) del Departamento de Educaclon de Estados Unidos para participar en 
un Importante estudio Internaclonal sobre el aprendizaje estudlantll. El nombre corto de este estudio es 
PIRLS; abrevlaclon de Estudio Internaclonal Sobre el Progreso en el Aprendizaje de la Lectura. Desde 
2001 PIRLS ha observado los logros de los estudlantes en lectura en las escuelas del mundo y cada 
cinco ahos documenta las tendenclas mundlales en el aprendizaje de la lectura. 

El resumen adjunto ofrece Informaclon de trasfondo de PIRLS, explIca lo que Implica la 
participaclon en el estudio para cada estudlante selecclonado e Incluye un numero de telefono y un 
correo electronico de contacto donde usted podra encontrar respuestas a cualquier pregunta que tenga. 

{Insert number} de nuestras clases de cuarto grade participaran. {Esta/Una de estas} es la clase 
de su hijo. {Esta clase/Estas dos clases}, junto con otras {500} clases de estudlantes de cuarto grade en 
todo el pais, contrlbulran a mostrar los logros en lectura de los estudlantes de cuarto grade en Estados 
Unidos, y como se comparan con estudlantes de cuarto grade alrededor del mundo. 

Para tener una Imagen precisa de lo que los estudlantes de cuarto grade en Estados Unidos 
pueden hacer, es Importante que cada estudlante selecclonado participe en el estudio. Lo Invito a que 
apoye este esfuerzo animando a su hIjo a participar; sin embargo, la participaclon en este estudio es 
completamente voluntarla. Las experlenclas anterlores parecen IndIcar que los estudlantes disfrutan de 
su participaclon. Ademas, los estudlantes que participen recibiran un pequeho regalo que creemos les 
gustara. 


Toda la Informaclon que se reuna es completamente confidenclal, como lo exige la ley. NCES 
esta autorizado a realizar este estudio de acuerdo con la Ley de reforma de clenclas de la educaclon de 
2002 (Ley pubilea 107-279, secclon 153). Por ley, la Informaclon dada por las escuelas, empleados y 
estudlantes solamente se puede utilizer con fines estadfsticos y no se puede dar a conocer ni utilizer de 
una manera que permita la Identificaclon de personas para otros fines (Ley pubIlea 107-279, secclon 183 
y tftulo V, subtftulo A de la Ley de goblerno electronico de 2002 (Ley pubilea 107-347)). En los Informes 
nunca se Identifica ni a los estudlantes ni a las escuelas. Todas las estadfsticas publicadas se refieren a 
Estados Unidos en conjunto. 

Antes de que podamos permitirle al niho participar en las actividades de PIRLS, debemos tener 
su autorizacion por escrito. Puede dejarnoslo saber completando el formulario que se adjunta 
(‘Participaclon en PIRLS’) y enviandolo a la escuela. 

Gracias por tomarse el tiempo de considerar este estudio. Reciba nuestros mejores deseos. 


Atentamente, 


Se adjuntan los siguientes documentos: 

Informaclon sobre PIRLS para los padres 
Participaclon en PIRLS 
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Exhibit C-12. PIRLS 2011 Expiicit Consent Letter - Continued 


PIRLS Explicit Consent Form 

Participacion en PIRLS 


Necesitamos su autorizacion. 

Nos encantaria que su hijo participara en el proyecto PIRLS junto con sus companeros de clase. La 
participacion de los estudiantes en PIRLS es sumamente importante para el exito del estudio y en ultima 
instancia para los intereses del pais de mejorar la educacion en lectura. Por favor marque "Si" a 
continuacion si da su autorizacion, o marque "No" si no la da. Si usted no desea que su hijo participe, la 
escuela se asegurara que su hijo realice otra actividad en el tiempo en que los demas estudiantes esten 
participando en PIRLS. Despues de marcar su eleccion, por favor firme el formulario y envielo a la escuela 
antes de {Insert date at least 2 days prior to assessment date} . 



Si, mi hijo puede participar en PIRLS. 



No, no deseo que mi hijo participe en el estudio. 


Nomhre del niho: 


Nombre complete 


Apellido 


Firma del padre o madre o guardian 


Fecha 


Gracias por tomarse el tiempo de considerar este importante proyecto. 
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Exhibit C-13. PIRLS 2011 Facts for Parents 


Main study’s ‘Facts About PIRLS for Parents’ - Grade 4 


Facts About PIRLS for Parents 

In April and May of this year, your child’s school will join schools across the U.S. and around the world 
taking part in PIRLS, the Progress in International Reading Literacy Study. The schools in the United States 
were selected randomly to represent our nation’s schools, and within each school, 4th-grade students were 
selected randomly to represent the nation’s 4th graders. Your child is among the 4th graders selected from 
this school to take part in PIRLS. 

What is PIRLS? 

PIRLS is an international assessment that measures student achievement in reading. Periodically (200 1, 
2006, and now 201 1) PIRLS documents worldwide trends in the knowledge of 4th graders. The National 
Center for Education Statistics within the U.S. Department of Education sponsors U.S. participation in 
PIRLS. Along with more than 50 other nations, we will take part in the 201 1 cycle just as we did in 2001 
and 2006. Participation in this study is voluntary. 

What is involved? 

Prom April through May 201 1, PIRLS staff will visit the school and administer an assessment that contains 
reading items. The assessment runs for 70 minutes with breaks between sections. Students will also receive 
a background questionnaire, which takes 20 to 30 minutes to complete. 

What are the benefits? 

The nation as a whole benefits from the contribution your child’s school makes to the national picture of the 
reading achievement of our 4th graders, and how they compare with 4th graders worldwide. Schools 
benefit too since we provide each school with a report about how it did in the assessment. Last, and 
certainly not least, students receive a small gift that we believe they will like. 

Who administers PIRLS? 

The entire assessment is administered by trained staff from Westat, a research organization under contract 
to the U.S. Department of Education’s National Center for Education Statistics. 


Where can I find out more about PIRLS? 

There is a lot of information available through the PIRLS website at http:/ / nces.ed.gov/ surveys/ PIRLS . 
Or, if you would like to contact a PIRLS staff member directly, please feel free to call the PIRLS hotline at 
I (888) 369-5033 or email us at TIMSS-PIRLS@westat.com. 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


C-33 


Exhibit C-13. PIRLS 2011 Facts for Parents - Continued 


Main study’s ‘Facts About PIRLS for Parents’ - Grade 4 

Informacion sobre PIRLS para los 
padres 

En abril y mayo de este ano, la escuela de su nino participara junto con otras escuelas de Estados Unidos y 
de todo el mundo en el Estudio Internacional Sobre el Progreso en el Aprendizaje de la Lectura, PIRLS, por 
sus siglas en ingles. Las escuelas fueron seleccionadas al azar para representar a las escuelas de nuestro pals, 
y dentro de cada escuela, se seleccionaron al azar a estudiantes de cuarto grado para representar a los 
estudiantes de cuarto grado del pals. Su nino es uno de los estudiantes de cuarto grado seleccionados de esta 
escuela para participar en PIRLS. 

iQue es PIRLS? 

PIRLS es una evaluacion internacional que mide el aprendizaje de los estudiantes en lectura. PIRLS 
documenta periodicamente (2001, 2006 y ahora en 201 1) las tendencias mundiales en el conocimiento de 
los estudiantes de cuarto grado. El Centro Nacional de Estadisticas de la Educacion dentro del 
Departamento de Educacion de Estados Unidos patrocina la participacion de Estados Unidos en PIRLS. 
Junto con mas de SO paises, participaremos en el ciclo del 20 1 1 tal como lo hicimos en el 200 1 y en el 
2006. La participacion en este estudio es voluntaria. 

^Que implica? 

Desde abril hasta mayo de 201 1 , el personal de PIRLS visitara la escuela y administrara una evaluacion que 
contiene temas de lectura. La evaluacion dura 70 minutos y tiene descansos entre las secciones. A los 
estudiantes tambien se les dara un cuestionario sobre informacion general, que toma de 20 a 30 minutos 
para responder. 

^Cuales son los beneficios? 

El pais en conjunto se beneficia de la contribucion que la escuela del nino hace para crear una imagen del 
pais sobre lo que los estudiantes de cuarto grado saben de lectura y como se comparan con los estudiantes 
de cuarto grado a nivel mundial. Las escuelas tambien se benefician ya que a cada una le damos un informe 
sobre cual fue su desempeno en la prueba. Finalmente, los estudiantes reciben un pequeno regalo que 
creemos les gustara. 

^Quien administra PIRLS? 

La totalidad de la evaluacion la administra personal capacitado de Westat, una compani'a de estudios de 
investigacion que tiene un contrato con el Centro Nacional de Estadisticas de la Educacion del 
Departamento de Educacion de Estados Unidos. 

Toda la informacion que se reuna sera completamente confidencial, como lo exige la ley. NCES esta 
autorizado a realizar este estudio de acuerdo con la Ley de reforma de ciencias de la educacion de 2002 (Ley 
publica 107-279, seccion 153). Bajo esa ley, la informacion dada por las escuelas, empleados y estudiantes 
solamente se puede utilizar con fines estadisticos y no se puede dar a conocer ni utilizar de una manera que 
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permita la identificacion de personas para otros fines (Ley publica 107-279, seccion 183 y ti'tulo V, subtitulo 
A de la Ley de gobierno electronico de 2002 (Ley publica 107-347)). En los informes nunca se identifica ni 
a los estudiantes ni a las escuelas. Todas las estadisticas publicadas se refieren a Estados Unidos en conjunto. 

^Donde puedo obtener mas informacion sobre PIRLS? 

Hay bastante informacion disponible en la pagina en Internet de PIRLS en http: / / nces.ed.gov/ surveys/ pirls/ . 
Si prefiere comunicarse con un empleado de PIRLS directamente, no dude en llamar a la linea de informacion 
de PIRLS al 1 (888) 369-5033 o enviarnos un correo electronico a TIMSSS-PIRLS(^westat.com. 
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PARENT GUARDIAN NOTIFICATION LETTER 
TIMSS 2011 Grade 8 


Dear Parent or Guardian, 

Our school has accepted an invitation to participate in an important assessment that will allow us to compare the 
mathematics and science performance of students in our state to students worldwide. The Trends in International 
Mathematics and Science Study (TIMSS) is conducted by the National Center for Education Statistics (NCES) within 
the U.S. Department of Education. TIMSS has documented student achievement worldwide in mathematics and 
science every four years since 1995, and this year a special study will be conducted to link TIMSS and the National 
Assessment of Educational Progress (NAEP). 

NAEP is also conducted by NCES and is the largest continuing and nationally representative assessment of what 
our nation's students can do in key subject areas such as mathematics, reading, science, and writing. Some 
students may be answering some NAEP questions during the TIMSS assessment. To access released NAEP 
questions, please visit the NAEP Questions Tool at http://nces.ed.gov/nationsreportcard/itmrlsx/default.aspx . 

The enclosed summary sheet provides some background to TIMSS, explains what is involved for each student 
selected to participate, and gives a contact phone number and e-mail address where you can find answers to any 
questions you might have. 

(Insert number) of our 8**^-grade classes will take part in TIMSS on (insert assessment date). (This/One of these) is 
your child's class. (This class/These two classes), along with some 1,200 other classes of 8*^-graders nationwide, 
will contribute to an understanding of what U.S. 8*^-graders know about mathematics and science and how they 
compare with 8‘^-graders worldwide. 

It is important that each student selected participates in the assessment to provide the most accurate report of 
what our students know and can do. The results are completely confidential, and your child's grade will not be 
affected. Your child may be excused from participation for any reason, is not required to finish the assessment, 
and may skip any test question. While TIMSS is voluntary, the results from this assessment will inform 
improvements in education and will allow states to compare their educational achievement in mathematics and 
science internationally. Your child will represent many other students, so participation is very important. 
However, if you do not want your child to participate, please notify me in writing by (date). 

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 9543). Under that law, the data provided by schools, staff, and students may be used only for statistical 
purposes and may not be disclosed, or used, in identifiable form for any other purpose (Public Law 107-279, 
Section 9573 and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347)). Students and schools are 
never identified in any reports. All reported statistics refer to the United States as a whole. 

We are excited that our school will be participating in TIMSS, and we are pleased that your child has been 
selected. We know that (school name)'s students will help us to show what our nation's students know and can 
do. 

Sincerely, 


Principal 


Enclosure: 

Facts About TIMSS for Parents 
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Exhibit C-14. TIMSS 2011 Eighth-Grade Notification Letter - Continued 
CARTA DE NOTIFICACION PARA LOS PADRES 0 TUTORES 
TIMSS 2011 Grade 8 


Estimado(a) padre o madre o tutor: 

Nuestra escuela ha aceptado una invitacion para participar en una importante evaluacion que nos permitira 
comparar el desempeno de los estudiantes de nuestro estado con estudiantes de todo el mundo en matematicas 
y ciencias. El Estudio de las Tendencias Internacionales de Matematica y Ciencias (TIMSS, por sus siglas en ingles) 
lo Neva a cabo el Centro Nacional para Estadfsticas de la Educacion (NCES, por sus siglas en ingles) dentro del 
Departamento de Educacion de Estados Unidos. TIMSS ha documentado los logros de los estudiantes en 
matematicas y ciencias en todo el mundo cada cuatro ahos desde 1995, y este ano se llevara a cabo un estudio 
especial para relacionar TIMSS y la Evaluacion Nacional de Progreso Educativo (NAEP, por sus siglas en ingles). 

NAEP tambien la Neva a cabo NCES y es la evaluacion continua y nacionalmente representativa mas grande que 
muestra lo que los estudiantes en toda la nacion saben y pueden hacer en el area de las materias fundamentales 
como matematicas, lectura, ciencias y escritura. Algunos estudiantes responderan algunas preguntas de NAEP 
durante la evaluacion de TIMSS. Para ver las preguntas publicadas de NAEP, por favor visite NAEP "Questions 
Tool" en http://nces.ed.gov/nationsreportcard/itmrlsx . 

La hoja de resumen que se adjunta proporciona alguna informacion sobre TIMSS, expNca lo que implica participar 
para cada estudiante seleccionado para participar y da un numero de telefono de contacto y una direccion de 
correo electronico (e-mail) donde usted puede encontrar respuestas a cualquier pregunta que tenga. 

(Insert number) de nuestras clases de 89 grado participaran en la evaluacion de TIMSS el (insert assessment date). 
(Esta/Una de estas) es la clase de su hijo(a). (Esta clase/Estas dos clases), junto con otras 1,200 clases de 
estudiantes de 89 grado en toda la nacion, contribuira(n) a una comprension de lo que los estudiantes de 89 grado 
en Estados Unidos saben acerca de matematicas y ciencias y como se comparan con los estudiantes de 89 grado 
de todo el mundo. 

Es importante que cada estudiante seleccionado participe en la evaluacion para proporcionar el informe mas 
precise posible de lo que los estudiantes saben y pueden hacer. Los resultados son completamente 
confidenciales, y no afectaran las notas de su hijo(a). Su hijo(a) puede ser excusado(a) de participar por cualquier 
motive, no esta obligado(a) a terminar la evaluacion y puede dejar de responder cualquier pregunta de la prueba. 
Aunque TIMSS es voluntaria, los resultados de esta evaluacion se tendran en cuenta en el memento de efectuar 
mejoras en la educacion y les permitiran a los estados comparar sus logros educativos en matematicas y ciencias 
internacionalmente. Su hijo(a) representara a muchos otros estudiantes, de manera que la participacion es muy 
importante. Sin embargo, si usted no quiere que su hijo(a) participe, por favor notifiqueme por escrito antes del 
(date). 

NCES esta autorizado bajo la Ley de Reforma de Ciencias de la Educacion de 2002 (Ley Publica 107-279, Seccion 
9543) para llevar a cabo este estudio. Bajo esa ley, la informacion proporcionada por las escuelas, el personal y los 
estudiantes se puede usar solamente para propositos estadisticos y no se puede publicar o usar de manera que 
pueda identificar a alguien para cualquier otro proposito (Ley Publica 107-279, Seccion 9573 y Titulo V, subtitulo A 
de la Ley de Gobierno Electronico de 2002 (Ley Publica 107-347)). Los estudiantes y las escuelas nunca se 
identifican en ningun informe. Todas las estadfsticas que se informan se refieren a Estados Unidos en conjunto. 

Estamos muy entusiasmados de que nuestra escuela participe en TIMSS y nos agrada que su hijo(a) haya sido 
seleccionado(a). Sabemos que los estudiantes de (school name) nos ayudaran a mostrar lo que los estudiantes 
de nuestra nacion saben y pueden hacer. 

Atentamente, 

Director(a) 

Se adjunta documento 

Informacion para los padres acerca de TIMSS 
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Exhibit C-15. TIMSS 2011 Eighth-Grade Facts for Parents 


Facts About TIMSS for Parents 

In April and May of this year, your child’s school will join schools across the United States and around the world 
taking part in TIMSS, the Trends in International Mathematics and Science Study. The schools were selected 
randomly to represent the nation’s schools, and within each school, 8*-grade students were selected randomly to 
represent the nation’s 8*-graders. Your child is among the 8*-graders selected from this school to take part in TIMSS. 


What is TIMSS? 

TIMSS is an international assessment that measures student learning in mathematics and science. Every four years 
since 1 995, TIMSS documents worldwide trends in the knowledge of 8*-graders. The National Center for Education 
Statistics within the U.S. Department of Education sponsors U.S. participation in TIMSS. Along with more than 60 
other nations, the U.S. will take part in the 201 1 cycle just as we did in 199S, 1999, 2003, and 2007. Participation in 
this study is voluntary. 

What is involved? 

Erom April through May 201 1 , TIMSS staff will visit the school and administer an assessment that contains 
mathematics and science items. The assessment runs for 90 minutes with breaks between sections. Students will also 
be asked some questions about themselves and their educational experience in a questionnaire that will take 20 to 30 
minutes to complete. 

What are the benefits? 

The nation as a whole benefits from the contribution your child’s school makes to the national picture of what our 8*- 
graders know about mathematics and science, and how they compare with 8*-graders worldwide. Schools benefit, 
too, since we provide each school with a report about how it performed in the assessment. Last, and certainly not 
least, students receive a small gift that we believe they will like. 

Who administers TIMSS? 

The entire assessment is administered by trained staff from Westat, a research organization under contract to the U.S. 
Department of Education’s National Center for Education Statistics. 

All of the information collected is kept completely confidential, as required by law. NCES is authorized to conduct 
this study under the Education Sciences Reform Act of 2002 (Public Law 107-279, Section 153). By law, the data 
provided by schools, staff, and students may be used only for statistical purposes and may not be disclosed or used in 
identifiable form for any other purpose (Public Law 107-279, Section 183 and Title V, subtitle A of the E- 
Government Act of 2002 (P.L. 107-347)). Students and schools are never identified in any reports. All reported 
statistics refer to select states and the United States as a whole. 

Where can I find out more about TIMSS? 

More information is available at the TIMSS website at http:/ /nces. ed.gov/timss/ or http ://timss.bc. edu / . Or, if 
you would like to contact a TIMSS staff member directly, please feel free to call the TIMSS hotline at 888-369-5033 
or email us at TIMSS-PIRLS@westat.com. 
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Informacion para las padres acerca de 
TIMSS 


En abril y mayo de este ano, la escuela de su hijo(a) se unira a otras escuelas en todo Estados Unidos y 
alrededor del mundo al tomar parte en TIMSS, el Estudio de las Tendencias Internacionales de 
Matematica y Ciencias. Las escuelas fueron seleccionadas al azar para representar a las escuelas de la 
nacion, y dentro de cada escuela se seleccionaron estudiantes de 8^ grade al azar para representar a los 
estudiantes de se grade de la nacion. Su hijo(a) esta dentro de los estudiantes de se grade seleccionados 
en esta escuela para participar en TIMSS. 

dQuees TIMSS? 

TIMSS es una evaluacion internacional que mide el aprendizaje de los estudiantes en matematicas y 
ciencias. Cada cuatro anos, desde 1995, TIMSS documenta en todo el mundo las tendencias en el 
conocimiento de los estudiantes de se grade. El Centro Nacional para Estadisticas de la Educacion dentro 
del Departamento de Educacion de Estados Unidos auspicia la participacion de Estados Unidos en 
TIMSS. Junto con mas de otras 60 naciones, Estados Unidos participara en el cicio 2011 tal come lo 
hicimos en 1995, 1999, 2003, y 2007. La participacion en este estudio es voluntaria. 

dQue implica? 

Desde abril hasta mayo de 2011, el personal de TIMSS visitara la escuela y administrara una evaluacion 
que contiene preguntas de matematicas y ciencias. La evaluacion dura 90 minutos con descansos entre 
las secciones. A los estudiantes tambien se les haran algunas preguntas acerca de si mismos y de sus 
experiencias educativas en un cuestionario que tomara 20 a 30 minutos para llenar. 

dCuales son los beneficios? 

Toda la nacion se beneficia de la contribucion que la escuela de su hijo(a) hace a la descripcion nacional 
de lo que nuestros estudiantes de se grado saben acerca de matematicas y ciencias, y como se 
comparan con los estudiantes de 8^ grado de todo el mundo. La escuela tambien se beneficia, ya que a 
cada escuela le proporcionamos un informe acerca de como les fue en la evaluacion. Finalmente, y 
ciertamente no menos importante, los estudiantes reciben un pequeno regalo que creemos que a ellos 
les gustara. 

dQuien administra TIMSS? 

Toda la evaluacion es administrada por personal capacitado de Westat, una compania que hace estudios 
de investigacion bajo contrato con el Centro Nacional para Estadisticas de la Educacion del 
Departamento de Educacion de Estados Unidos. 

Toda la informacion que se reuna se mantendra en complete confidencialidad como lo exige la ley. NCES 
esta autorizado bajo la Ley de Reforma de Ciencias de la Educacion de 2002 (Ley Publica 107-279, 

Seccion 153) para Never a cabo este estudio. Por ley, la informacion proporcionada por las escuelas, el 
personal y los estudiantes se puede user solamente para propositos estadisticos y no se puede publicar 
o user de manera que pueda identificar a alguien para cualquier otro proposito (Ley Publica 107-279, 
Seccion 9573 y Titulo V, subtitulo A de la Ley de Gobierno Electronico de 2002 (Ley Publica 107-347)). 
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Los estudiantes y las escuelas nunca se identifican en ningun informe. Todas las estadisticas que se 
informan se refieren a Estados Unidos en conjunto. 

dDonde puedo encontrar mas informacion acerca de TIMSS? 

Hay mas informacion disponible en el sitio Web de TIMSS en http://nces.ed.gov/timss/ o 
http://timss.bc.edu/ . Si usted desea comunicarse directamente con un miembro del personal de TIMSS, 
por favor no demore en llamar a la linea directa al 888-369-5033 o mandenos un correo electronico (e- 
mail) a TIMSS-PIRLS@westat.com. 
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Timeline of Activities 


Progress in International 
Reading Literacy Study 
(PIRLS) 

2011 



For questions about TIMSS-PIRLS 2011, 
oontact theTIMSS-PIRLS Information Hotline 
at 1-888-369-5033 
or email TIMSS-PIRLS@westat.oom 



les 


NATIONAL CENTER rod 
EDUCATION STATISTICS 



Trends in 
International 
Mathematics 
and Science Study 
(TIMSS) 

2011 


Progress in International 
Reading Literacy Study 
(PIRLS) 

2011 



Exhibit D-1. TIMSS and PIRLS timeline of aetivities — Continued 
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TIMELINE OF SCHOOL ACTIVITIES 
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Prior to assessment day 
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Principal 

School coordinator 

Teachers of 
sompled classes 

■ Identifies a school coordinator. The school 
coordinator works with Westat assessment 
staff to plan for the assessment. 

■ Completes a brief School Questionnaire (about 
the characteristics of the school; its enrollment, 
resources, policies, and learning environment). 

■ Arranges assessment day space. 

■ Confirms space for assessment 
is problem-free. 

■ If necessary, helps to ensure all sampled 
students attend the assessment session. 

■ Receives a $200 check for the school. 

■ Represents other similar U.S. schools. 

■ Receives feedback based on the 
performance of students in your school 
that took the TIMSS/PIRLS assessment. 

■ Receives U.S. national report with 
final results. 
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■ Works with Westat assessment staff to select 
an assessment date convenient for the school. 

■ Completes Class Listing Form and Student 
Listing Forms and returns these to Westat 
(via fax, mail, or email). 

■ Ensures parents are notified that their children 
have been selected for the assessment. 

■ Works with assessment staff to identify 
students with special education needs. 

■ Meets with students/teachers as necessary 
to provide information about the study. 

■ Collects completed School and Teacher 
Questionnaires and returns them to assessment 
staff. 

■ Ensures all sampled students attend the 
assessment session. 

■ Meets with assessment staff and reviews 
the assessment. 

■ Receives a $100 personal check. 

■ Receives U.S. national report with 
final results. 

u 

o. 
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■ Complete Teacher Questionnaires and returns 
them to the school coordinator prior 
to assessment day. 


■ Represent the United States 
in the international study. 

o 

o> 

Students 





■ Students of the selected classes attend 

■ Receive a small thank-you gift. 

o 




the assessment session and complete the 

■ Represent other U.S. students like 

o 

E 




assessment and Student Questionnaire. 

themselves and contribute to the profile 

"c 





of what American students know. 

"5 

c 

Westot assessment staff 

■ Work with the school to set an assessment date. 

■ Call the school coordinator to discuss assess- 

■ Conduct assessment from start to finish. 




(contracted by the U.S. Department 

■ Help school coordinator with assessment details. 

ment day space and student participation. 

■ Furnish all the assessment materials, pencils. 


o 

of Education's National Center 
for Education Statistics to 

■ Protect school and student confidentiality. 

■ Select classroom sample and notify school 

and test booklets. 



conduct the study and support 


of selected classes. 

■ Conduct a brief followup Interview with the 



participating schools) 


■ Provide School and Teacher Questionnaires 

school coordinator at the end of the assessment. 





to the school coordinator for distribution. 

■ Maintain security of all materials. 
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Exhibit D-2. TIMSS & PIRLS summary of activities for school coordinator 



Summary of Activities 
for Schooi Coordinators 

Grade 4 


PIRLS 


What will be asked of the school coordinator? 

Upon the school's agreement to participate, Westat staff 

will work with the school coordinator to: 

■ Schedule the assessment. A Westat staff member 
will contact the school coordinator to schedule a 
convenient date between April 4 and May 27, 201 1. 

The coordinator will need to arrange the use of each 
selected class’ classroom or an alternative quiet space 
for the assessment. 

■ Review parent notification procedures. If your 
school requires parental permission to conduct the 
assessment(s), the Westat staff member will review 
these procedures with the school coordinator. 

■ Provide a list of fourth-grade classes. The school 
coordinator will receive instructions for preparing and 
submitting a list of fourth-grade classes. Classes from 
the list will be selected randomly to participate. 

■ Provide a student listing for each selected class. 

All student names will be kept confidential and will 
never be linked to assessment bookiets or resuits. 
Individuai student responses or scores are NEVER 
reported or distributed. 

Closer to the assessment date, the school coordinator 

will be asked to: 

■ Work with a Westat staff member to identify those 
students with special needs or limited English 
proficiency that preclude them from participating 
in the assessment. 


■ Notify parents, teachers, and students. Once the classes 
and students have been selected, a Westat staff member 
will work with the school coordinator on procedures 
for notifying parents, teachers, and students of the 
study and the benefits of participating. 

■ Receive the School and Teacher Questionnaires. 

The school coordinator will be mailed the School 
and Teacher Questionnaires and asked to distribute 
them to the school principal and teachers of the 
selected classes. The school coordinator should also 
retrieve the questionnaires and return them to the 
Westat staff member on assessment day. 

■ Confirm the assessment information. At least 2 weeks 
before the assessment, a Westat staff member will 
contact the school coordinator to confirm the date 
and location of the assessment. 

On assessment day, the school coordinator will be asked to: 

■ Ensure that all students in the selected classes attend 
the assessment session. While it is not necessary for 
the school coordinator to be present during the session, 
the school coordinator should be available before 

the assessment to help locate selected students and 
ensure participation. It is very Important that student 
attendance rates be as high as possibie to avoid 
the need for a makeup session. 

A graphic timeline of activities is also available for 

your convenience. 


Please feel free to contact the U.S. TIMSS-PIRLS Home Office with any questions 
via email at TiMSS-PiRLS@westat.com 
or by calling 1 - 888 - 369-5033 



O.M.B. No. 1850-0645 
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Exhibit D-3. TIMSS & PIRLS FAQ 



Frequently Asked Questions 

Grade 4 


PIRLS 


What is TIMSS and PIRLS? 

TIMSS and PIRLS are international assessments designed to 
measure trends in mathematics and science achievement (TIMSS) 
and reading literacy (PIRLS) of fourth-grade students. In 20 1 1 , 
the two studies will be administered simultaneously in participating 
schools. Up to four classes will be randomly selected to participate, 
with each class randomly assigned to take either TIMSS or PIRLS. 

Why was my school selected for participation? 

Schools of varying demographics and locations were randomly 
selected so that the overall U.S. sample is representative of the overall 
U.S. school population. The random selection process is important 
for ensuring that a country’s sample accurately reflects its schools 
and, therefore, can fairly be compared with samples of schools from 
other countries. 

Will all our fourth-graders be asked 
to participate? 

It depends on the number of fourth-grade classrooms in the school. 
In schools with four or fewer fourth-grade classrooms, all students 
will be asked to participate. In schools with more than four fourth- 
grade classrooms, only students in four randomly selected classrooms 
will be asked to participate. 

In addition, some students with special needs or limited English 
proficiency may be excused from the assessment. 

Who conducts the assessment? 

The entire assessment process will be undertaken by trained staff 
from Westat, a research organization under contract to the U.S. 
Department of Education’s National Center for Education Statistics 
(NCES). NCES conducts this study under authorization in the 
Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). The U.S. Office of Management and Budget has 
approved the data collection under OMB # 1850-0645. 

Do teachers need to help administer 
the assessment? 

No. Westat staff will visit the school on the day of the assessment, 
bringing with them all the materials required, and they will handle 
the entire administration of the assessment. 

When will the assessment be conducted? 

The assessment will be conducted between April 4 and May 27, 
2011. Westat will work with schools to identify an assessment 
date convenient for the school in that time period. 


Where will the assessment be conducted? 

The assessment will be conducted in the schools that are selected 
to participate. 

How long does the assessment take? 

The assessment session is approximately 2 V 2 hours and includes the 
administration of the assessment, a brief questionnaire that students 
complete about themselves, and two breaks. The questionnaire takes 
approximately 30 minutes to complete. 

What will happen with the collected data? 

The data from the assessment will be used to evaluate how the 
knowledge and skills of U.S. students compare to those of their 
peers in other participating countries. By law, the data provided by 
schools, staff, and students may be used only for statistical purposes 
and may not be disclosed, or used, in identifiable form for any other 
purpose (Public Law 107-279, Section 183 and Title V, subtitle 
A of the E-Government Act of 2002 (PL. 107-347)). Reports of 
the findings from the assessment will not identify participating 
districts, schools, students, or individual staff Individual responses 
will be combined with those from other participants to produce 
summary statistics and reports. 

Are schools required by federal law 
to participate? 

No. School participation is voluntary. However, we hope you will 
participate in this study so that students like those in your school 
are accurately and fairly represented. 

What do school staft and students do? 

■ Schools are asked to designate a School Coordinator to assist 
Westat staff members with in-school arrangements. 

■ The school principal or lead administrator will receive a 
School Questionnaire to complete. The questionnaire, which 
asks about the school environment and structure, takes about 
20 minutes to complete. 

■ Teachers of the sampled fourth-grade classes will receive Teacher 
Questionnaires to complete. These questionnaires focus on the 
nature of implemented curricula, instructional practices, and 
attitudes toward reading, mathematics, and science. They take 
about 30 minutes to complete. 

■ Students will attend the assessment session [2Vi hours in 
length) and will be asked to complete the assessment booklet 
and questionnaire. Those who do will receive a small gift as 

a thank-you. 


Please feel free to contact the U.S. TIMSS— PIRLS Home Office 
with any questions via email TIMSS-PIRLS@westat.eom or 
by calling 1 - 888 - 369 - 5033 . 


O.M.B. No. 1850-0645 
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Participating countries in PIRLS 201 1 


North and 
South America 

Canada 

Colombia 

Honduras 

Trinidad and Tobago 
United States 

Europe 

Austria 

Belgium (French) 
Bosnia and 
Herzegovina 
Bulgaria 
Croatia 

Czech Republic 

Denmark 

England 

Finland 

France 

Georgia 

Germany 

Hungary 


Ireland 

Indonesia 

Italy 

Iran, Islamic Rep. of 

Lithuania 

Israel 

Luxembourg 

Kuwait 

Malta 

Mongolia 

Netherlands 

Oman 

Norway 

Qatar 

Poland 

Saudi Arabia 

Romania 

Singapore 

Russian Federation 

United Arab 

Scodand 

Emirates 

Slovak Republic 

Africa 

Slovenia 

Botswana 

Spain 

Sweden 

Ukraine 

Egypt 

Libya 

Morocco 

Asia and 

South Africa 

Middle East 

Armenia 

Australia 
and Oceania 

Azerbaijan 

Australia 

Chinese Taipei 
Hong Kong SAR 

New Zealand 


Benchmarking participants 

Abu Dhabi, UAE Dubai, UAE Quebec, Canada 

Alberta, Canada Ontario, Canada 


PIRLS assesses reading for literary 
experience and for acquiring and using 
information. PIRLS asks students to 
engage in a full repertoire of reading 
skills and strategies, including: 

• Focusing on and retrieving explicitly 
stated information; 

• Making straightforward inferences; 

• Interpreting and integrating ideas 
and information; and 

• Examining or evaluating content, 
language, and textual elements. 


NCES is authorized to conduct PIRLS 
under Section 1 53, of Public Law 1 07-279. 
Information collected will help the U.S. 
Department of Education’s ongoing efforts 
to benchmark student achievement in 
the United States. Participation is voluntary. 
Data collected may be used only for 
statistical purposes and may not be 
disclosed, or used, in identifiable form for 
any other purpose (Title V subtitle A of the 
E-Government Act of 2002 (RL. 1 07-347) 
and Section 183, Public Law 107-279). 

The U.S. Office of Management and Budget 
has approved the data collection under 
OMB # 1 850-0645. Individual responses 
will be combined with those from other 
participants to produce summary statistics 
and reports. 


PIRLS 



For questions about PIRLS 201 1 , 
contact the PIRLS Information Hotline 
at 1-888-369-5033 
or email TIMSS-PIRLS@westat.com 



Progress in 
International 
Reading Literacy 
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What is PIRLS? 

The Progress in International Reading Literacy Study 
(PIRLS) is an international assessment and research 
project designed to measure both trends in fourth- 
grade students’ reading literacy achievement as well 
as school and teacher practices related to reading 
instruction. PIRLS 2011 is the third such study 
in the PIRLS series of internationally comparative 
reading studies carried out in countries around the 
world every 5 years. In PIRLS 2011, students from 
more than 50 countries, including the United States, 
will participate. 

PIRLS is sponsored by the International Association 
for the Evaluation of Educational Achievement 
(lEA) and managed in the United States by the 
National Center for Education Statistics (NCES), 
part of the U.S. Department of Education. 

Why is PIRLS important? 

PIRLS provides a unique opportunity to compare 
the reading knowledge and skills of U.S. fourth- 
graders with their peers in countries around 
the world. PIRLS complements what we learn 
from national assessments by identifying the 
strengths and weaknesses of students in reading 
relative to students around the world. The results 
inform national discussions about U.S. education 
performance and practice within the wider 
context of international 
competitiveness. 

Moreover, by 
participating in 
PIRLS 2011, the 
United States will 
obtain data about 
changes in children’s 



reading achievement over the past 10 years, 
including valuable information about changes 
in reading instruction, how those changes 
relate to students’ performance in reading, and 
about home, school, and classroom influences 
on reading achievement. 

What type ot reading 
assessment is PIRLS? 

PIRLS is designed to reflect the reading curriculum 
used in participating countries. PIRLS asks students 
to read two texts, either two literary texts (narrative 
fiction, generally drawn from children’s books), 
two informational texts (typically excerpts from 
biographies, step-by-step instructions, or scientific 
or non-fictional materials), or one of each type. 

It then asks students about a dozen questions — 
both multiple-choice and open-ended “constructed 
response” questions — about the texts. These 
questions may range from identifying the place, 
time, and actions of the main characters or events 
to interpreting how characters might feel, why 
events occurred, or what the passage means 
overall (e.g., does the story teach a lesson?). 
Examples of released PIRLS test items can be viewed 
at http://nces.ed.gOv/pubs2008/2008017_2.pdf. 


Key findings 
from PIRLS 2006 

• In PIRLS 2006, the average U.S. 4th- 
grader's reading literacy score (540) 
\A/as above the PIRLS scale average 
of 500 but below that of 4th-graders 
in 7 of the other 39 participating 
countries and 3 of the 5 participating 
Canadian provinces. 

• Among the 28 countries that 
participated in both the 2001 and 
2006 PIRLS assessments, the average 
reading literacy score increased in 8 
countries and decreased in 6 countries. 
In the rest of the countries, including 
the United States, there was no 
measurable change in the average 
reading literacy score. 


PIRLS is also meant to study home and school 
factors associated with children’s reading literacy 
by the fourth grade. To that end, PIRLS will 
also administer questionnaires to students, their 
teachers, and the principals of their schools. The 
questions are designed to measure key aspects of 
students’ home and school environments. In this 
way, PIRLS provides each country with a rich 
source of information on the factors influencing 
reading literacy. 
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For questions aboutTIMSS 201 1 , 
contact the TIMSS Information Hotline 
at 1-888-369-5033 
or emaii TiMSS-PIRLS@westat.com 




Exhibit D-5. TIMSS eighth-grade timeline of activities — Continued 
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TIMELINE OF SCHOOL ACTIVITIES 



After agreement to participate 

Priar to assessment day 

Assessment day 

Benefits 


Principal 

■ Identifies a school coordinator. The school 

■ Completes a brief School Questionnaire (about 

■ Confirms space for assessment 

■ Receives a $200 check for the school . 



coordinator works with Westat assessment 

the characteristics of the school, its enrollment. 

is problem-free. 

■ Represents other similar U.S. schools. 



staff to plan for the assessment. 

resources, policies, and learning environment). 

■ If necessary, helps to ensure all sampled 

■ Receives feedback based on the 




■ Arranges assessment day space. 

students attend the assessment session. 

performance of students in your school 
that took the TIMSS assessment. 






■ Receives U.S. national report with 






final results. 

U) 

Vi 

School coordinator 

■ Works with Westat assessment staff to select 

■ Completes Class Listing Form and Student 

■ Collects completed School and Teacher 

■ Receives a $100 personal check. 



an assessment date convenient for the school. 

Listing Forms and returns these to Westat 

Questionnaires and returns them to 

■ Receives U.S. national report with 




(via fax, mail, or email). 

assessment staff. 

final results. 




■ Ensures parents are notified that their children 

■ Ensures all sampled students attend 


o> 



have been selected for the assessment. 

the assessment session. 


0) 

u 



■ Works with assessment staff to Identify 

■ Meets with assessment staff and reviews 





students with special education needs. 

the assessment. 





■ Meets with students/teachers as necessary 



o 



to provide information about the study. 



O) 

c 

Teachers of 



■ Complete Teacher Questionnaires and 



■ Represent the United States 


sampled classes 


returns them to the school coordinator prior 


in the international study. 

E 

"c 

"o 

c 

o 



to assessment day. 



•a 

Students 





■ Students of the selected classes attend 

■ Receive a small thank-you gift. 





the assessment session and complete the 

■ Represent the United States 

LU 




assessment and Student Questionnaire. 

in the international study. 


Westat assessment staff 

■ Work with the school to set an assessment date. 

■ Call the school coordinator to discuss 

■ Conduct assessment from start to finish. 

— 


(contracted by the U.S. Department 

■ Help school coordinator with assessment details. 

assessment day space and student 

■ Furnish all the assessment materials, pencils. 



ot Education's National Center 

■ Protect school and student confidentiality. 

participation. 

and test booklets. 



for Education Statistics to 
conduct the study and support 
participating schools) 

■ Select classroom sample and notify school 
of selected classes. 

■ Conduct a brief followup interview with the 
school coordinator at the end of the assessment. 




■ Provide School and Teacher Questionnaires 

■ Maintain security of all materials. 





to the school coordinator for distribution. 
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Exhibit D-6. TIMSS eighth-grade summary of activities for school coordinator 



Summary of Activities 
for Schooi Coordinators 

Grade 8 


What will be asked of the school coordinator? 

Upon the school's agreement to participate, Westat staff 

will work with the school coordinator to: 

■ Schedule the assessment. A Westat staff member 
will contact the school coordinator to schedule a 
convenient date between April 4 and May 27,2011. 

The coordinator will need to arrange the use of each 
selected class’ classroom or an alternative quiet space 
for the assessment. 

■ Review parent notification procedures. If your 
school requires parental permission to conduct the 
assessment(s), the Westat staff member will review 
these procedures with the school coordinator. 

■ Provide a list of eighth-grade mathematics classes. 

The school coordinator will receive instructions 
for preparing and submitting a list of eighth-grade 
mathematics classes. Classes from the list will be 
selected randomly to participate. 

■ Provide a student listing for each selected class. 

All student names will be kept contidential and will 
never be linked to assessment booklets or results. 
Individual student responses or scores are NEVER 
reported or distributed. 

Closer to the assessment dale, the school coordinator 

will be asked to: 

■ Work with a Westat staff member to identify those 
students with special needs or limited English 
proficiency that preclude them from participating 
in the assessment. 


■ Notify parents, teachers, and students. Once the classes 
and students have been selected, a Westat staff member 
will work with the school coordinator on procedures 
for notifying parents, teachers, and students of the 
study and the benefits of participating. 

■ Receive the School and Teacher Questionnaires. 

The school coordinator will be mailed the School 
and Teacher Questionnaires and asked to distribute 
them to the school principal and teachers of the 
selected classes. The school coordinator should also 
retrieve the questionnaires and return them to the 
Westat staff member on assessment day. 

■ Confirm the assessment information. At least 2 weeks 
before the assessment, a Westat staff member will 
contact the school coordinator to confirm the date 
and location of the assessment. 

On assessment day, the school coordinator will be asked to: 

■ Ensure that all students in the selected classes attend 
the assessment session. While it is not necessary for 
the school coordinator to be present during the session, 
the school coordinator should be available before 

the assessment to help locate selected students and 
ensure participation. It is very important that student 
attendance rates be as high as possible to avoid 
the need for a makeup session. 

A graphic timeline of activities is also available for 

your convenience. 


Please feel free to contact the U.S. TIMSS Home Office with any questions 
via email at TIMSS-PIRLS@westat.com 
or by calling 1 - 888 - 369-5033 


O.M.B. No. 1850-0645 
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Frequently Asked Questions 

Grade 8 


What is the TIMSS? 

TIMSS is an international assessment designed to measure trends in 
mathematics and science achievement of U.S. eighth-grade students. 
It provides a benchmark for how U.S. eighth-grade students perform 
relative to their peers in other countries. TIMSS also collects data 
from principals and teachers about school conditions and practices 
that relate to student achievement. TIMSS was first administered in 
1995 and is in its fifth cycle. In 2011, more than 60 countries will 
participate in TIMSS. 

Why was my school selected for participation? 

Schools of varying demographics and locations were randomly 
selected so that the overall U.S. sample is representative of the overall 
U.S. school population. The random selection process is important 
for ensuring that a country’s sample accurately reflects its schools 
and, therefore, can fairly be compared with samples of schools from 
other countries. 

Will all our eighth-graders be asked 
to participate? 

It depends on the number of eighth-grade classrooms in the school. 
In schools with only one or two eighth-grade classrooms, all students 
will be asked to participate. In schools with more than two eighth- 
grade classrooms, only students in two randomly selected classrooms 
will be asked to participate. 

In addition, some students with special needs or limited English 
proficiency may be excused from the assessment. 

Who conducts the assessment? 

The entire assessment process will be undertaken by trained staff 
from Westat, a research organization under contract to the U.S. 
Department of Education’s National Center for Education Statistics 
(NCES). NCES conducts this study under authorization in the 
Education Sciences Reform Act of 2002 (Public Law 107-279, 
Section 153). The U.S. Office of Management and Budget has 
approved the data collection under OMB # 1850-0645. 

Do teachers need to help administer the 
assessment? 

No. Westat staff will visit the school on the day of the assessment, 
bringing with them all the materials required, and they will handle 
the entire administration of the assessment. 

When will the assessment be conducted? 

The assessment will be conducted between April 4 and May 27, 
2011. Westat will work with schools to identify an assessment 
date convenient for the school in that time period. 


Where will the assessment be conducted? 

The assessment will be conducted in the schools that are selected 
to participate. 

How long does the assessment take? 

The assessment session is approximately 2 Vi hours and includes 
the administration of the assessment, a brief questionnaire 
that students complete about themselves, and two breaks. 

The questionnaire takes approximately 30 minutes to complete. 

What will happen with the collected data? 

The data from the assessment will be used to evaluate how the 
knowledge and skills of U.S. students compare to those of their 
peers in other participating countries. By law, the data provided by 
schools, staff, and students may be used only for statistical purposes 
and may not be disclosed, or used, in identifiable form for any other 
purpose (Public Law 1 07-279, Section 1 83 and Title V, subtitle 
A of the E-Government Act of 2002 (P.L. 107-347)). Reports of 
the findings from the assessment will not identify participating 
districts, schools, students, or individual staff. Individual responses 
will be combined with those from other participants to produce 
summary statistics and reports. 

Are schools required by federal law 
to participate? 

No. School participation is voluntary. However, we hope you will 
participate in this study so that students like those in your school 
are accurately and fairly represented. 

What do school staff and students do? 

■ Schools are asked to designate a School Coordinator to assist 
Westat staff members with in-school arrangements. 

■ The school principal or lead administrator will receive a School 
Questionnaire to complete. The questionnaire, which asks about 
the school environment and structure, takes about 20 minutes 
to complete. 

■ Teachers of the sampled eighth-grade classes will receive Teacher 
Questionnaires to complete. These questionnaires focus on the 
nature of implemented curricula, instructional practices, and 
attitudes toward mathematics and science. They take about 30 
minutes to complete. 

■ Students will attend the assessment session {IVi hours in 
length) and will be asked to complete the assessment booklet 
and questionnaire. Those who do will receive a small gift as 

a thank-you. 


For questions about TIMSS 2011, contact the TIMSS Information 
Hotline at 1-888-369-5033 or email TIMSS-PIRLS@westat.COm. 



O.M.B. No. 1850-0645 
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Exhibit D-8. TIMSS eighth-grade study broehure 


Other information 
coiiected by TIMSS 

TIMSS is more than an assessment of student 
knowledge in mathematics and science. TIMSS 
also considers the context in which learning 
occurs. Students, teachers, and schools are asked 
about a variety of aspects of the environments in 
which content is taught, learned, practiced, and 
applied. In this way, TIMSS provides each country 
with a rich source of information on the factors 
influencing mathematics and science achievement. 

Participating countries in TIMSS 2011 


North and 

Russian Federation 

Mongolia 

South America 

Scotland 

Oman 

Chile 

Serbia 

Palestinian Nat’l 

Honduras 

Slovak Republic 

Auth. 

United States 

Slovenia 

Qatar 

Europe 

Spain 

Saudi Arabia 

Austria 

Sweden 

Singapore 

Belgium (Flemish) 

Turkey 

Syrian Arab 

Bosnia and 

Ukraine 

Republic 

Herzegovina 

Asia and 

Thailand 

Czech Republic 

Middle East 

United Arab 

Denmark 

Armenia 

Emirates 

England 

Azerbaijan 

Yemen 

Finland 

Bahrain 

Africa 

Georgia 

Chinese Taipei 

Botswana 

Germany 

Hong Kong SAR 

Egypt 

Hungary 

Indonesia 

Ghana 

Ireland 

Iran, Islamic Rep. of 

Libya 

Italy 

Israel 

Morocco 

Lithuania 

Japan 

South Africa 

Malta 

Jordan 

Tunisia 

Macedonia 

Kazakhstan 


Netherlands 

Korea, Rep. of 

Australia 
and Oceania 

Norway 

Kuwait 

Australia 

Poland 

Lebanon 

New Zealand 

Romania 

Malaysia 


Benchmarking participants 


NCES is authorized to conduct TIMSS 
under Section 153, of Pubiic Law 107-279. 
Information coiiected will help the U.S. 
Department of Education's ongoing efforts 
to benchmark student achievement in 
the United States. Participation is voluntary. 
Data collected may be used only for 
statistical purposes and may not be 
disclosed, or used, in identifiable form for 
any other purpose (Title V, subtitle A of the 
E-Government Act of 2002 (RL. 107-347) 
and Section 183, Public Law 107-279). 

The U.S. Office of Management and Budget 
has approved the data collection under 
OMB # 1 850-0645. Individual responses 
will be combined with those from other 
participants to produce summary statistics 
and reports. 




• IW4#E 


For questions about TIMSS 201 1 , 
contact the TIMSS Information Hotline 
at 1-888-369-5033 
or email TIMSS-PIRLS@westat.com 
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What is TIMSS? 

The Trends in International Mathematics and 
Science Study (TIMSS) is an international 
assessment and research project designed to measure 
trends in mathematics and science achievement 
at the fourth- and eighth-grade levels as well as 
school and teacher practices related to instruction. 
Since 1995, TIMSS has been administered every 
4 years. TIMSS 2011, the fifth study in the series, 
will involve students from more than 60 countries, 
including the United States. 

TIMSS is sponsored by the International Association 
for the Evaluation of Educational Achievement 
(lEA) and managed in the United States by the 
National Center for Education Statistics (NCES), 
part of the U.S. Department of Education. 

Why is TIMSS important? 

TIMSS provides a unique opportunity to compare 
U.S. students’ math and science knowledge 
and skills at the fourth- and eighth-grade levels 
with that of their peers in countries around the 
world. TIMSS complements what we learn from 
national assessments by identifying the strengths 
and weaknesses of student performance relative 
to students around the world. The results inform 
national discussions about education as well as 
international competitiveness. 




TIMSS provides valuable benchmark information 
on how U.S. students compare to students around 
the world, allows educators and policymakers to 
examine other educational systems for practices 
that could have application to the United States, 
and contributes to ongoing discussions of ways to 
improve the quality of education for all students. 

What type of assessment 
is TIMSS? 

The TIMSS mathematics and science assessment 
is developed through an international consensus- 
building process involving input from U.S. and 
international experts in mathematics, science, and 
measurement. In a final step, the assessment is 
endorsed as suitable by all participating countries. 
The assessment contains a mix of questions: some 
require students to select appropriate responses, 
while others require that students solve problems 
and provide written answers. Examples of released 
TIMSS items are available at http://nces.ed.gov/ 
timss/educators.asp. 


Key findings 
from TIMSS 2007 

• In TIMSS 2007, the average U.S. 4th- 
grader's mathematics score (529) 
was above the TIMSS scale average 
of 500 but below that of 4th-graders 
In 8 of the other 35 participating 
countries. The average U.S. 8th-graders' 
mathematics score (508) was alsa 
above the TIMSS scale average (500) 
but below that of 8th-graders in 5 of 
the other 47 participating countries. 

At both 4th and 8th grades, U.S. math 
scores in 2007 were higher than 

in 1995. 

• In science, the 2007 TIMSS average 
U.S. 4th-graders' science score 
(539) was above the TIMSS scale 
average (500) but below that 

of 4th-graders in 4 of fhe ofher 35 
participating countries. The average 
U.S. 8th-grader's science score 
(520) was also above the TIMSS 
scale average (500) but below that 
of 8fh-graders in 9 of the other 47 
participating countries. At both 4th 
and 8th grade, U.S. science scores 
in 2007 were not measurably different 
than in 1995. 
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Exhibit E- 1 . T est administrator training agenda 


TIMSS-PIRLS 2011 
Test Adnninistrator Training Agenda 
March 18 - 19, 2011 - Los Angeles, CA 



DAY 1: 8:30 AM - 5:30 PM 

Session #1: Introduction 

8:30 - 9:00 

• Introduction to TIMSS - PIRLS 

to TIMSS & PIRLS 2011 


• Timeline of Activities 

• TA Role Overview 

Session #2: Checking 

9:00 - 9:45 

• Review Assignment Materials 

Assignments 


• Review School Folder Content 

Session # 3: Field 
Management System 

9:45-10:15 

• Review of the FMS 


10:15 - 10:30 Morning Break 

Session #4: Test Materials 

10:30 - 12:00 

• Review of Session Materials 

Demo 


• Review of Bulk Materials 


12:00 - 

1:00 Lunch 

Session #5: Pre- 

1:00 - 2:00 

• Review Preassessment Activities 

assessment Activities 


• Review Student Tracking Forms 

• Review Questionnaire Status 

Session #6: Pre- 

2:00 - 3:00 

• Role Play calling First Middle or Lincoln 

assessment Call Role Play 
# 1 and 2 


Academy 


3:00 - 3:15 Afternoon Break 

Session #7: FMS Entry 

3:15 - 4:00 

• Entering Preassessment Call Information 
into the FMS - First Middle or Lincoln 
Academy 

Session #8: Pre- 

4:00 - 5:00 

• Role - Play calling St. Peter Catholic or 

assessment Call Role Play 
# 3 and 4 


Jacobsen Academy 

Session #9: FMS Entry 

5:00 - 5:30 

• Entering Preassessment Call Information 
into the FMS - St. Peter Catholic or 
Jacobsen Academy 


2/28/2011 
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Exhibit E- 1 . T est administrator training agenda — Continued 


TIMSS-PIRLS 2011 
Test Administrator Training Agenda 
March 18 - 19, 2011 - Los Angeles, CA 



DAY 2: 

8:30 

- 5:30 

Session #10: Activities 

8:30 - 9:00 

• 

Activities Upon Arrival 

Upon Arrival at the School 


• 

Materials to Bring to the School 

Session #11: Preparing 

9:00 - 9:45 

• 

Review Steps for Preparing Booklets 

the Booklets 


• 

Review Manipulatives 



• 

Coding Students 

Session #12: Preparing 

9:45 - 10:15 

• 

Exercise #1: Preparing the Booklets for a 

the Booklets - Exercise #1 



TIMSS 8 Class 

10:15 - 10:30 Morning Break 

Session #13: Preparing 

10:30-11:00 

• 

Exercise #2: Preparing the Booklets for a 

the Booklets - Exercise #2 



TIMSS-NAEP Link Class 

Session #14: Conducting 

11:00-12:00 

• 

Arrival at the School 

the TIMSS/PIRLS 


• 

Distributing Booklets 

Assessment According to 


• 

Reading the Script 

International Guidelines 





12:00 - 

1:00 Lunch 

Session #15: Session 

1:00 - 1:30 

• 

Role Play Reading the Session Script for a 

Script Dyad 



TIMSS 8 and TIMSS-NAEP Link Session 

Session #16: 

1:30 - 2:00 

• 

Review of the Questionnaire Guide 

Questionnaire Guide 


• 

How to Answer Student Questions 

Session #17: Supervising 

2:00 - 2:30 

• 

Review How to Supervise Each Session 

the Session 



Type: TIMSS, PIRLS, TIMSS-NAEP Link 



• 

Supervising the Use of Manipulatives 



• 

Handling Difficult Situations 

Session #18: Concluding 

2:30 - 3:30 

• 

Ending the Session 

the Session 


• 

Collecting Booklets 



• 

Activities to Complete Before Leaving the 




School 

3:30 - 3:45 Afternoon Break 

Session #19: Training 

3:45 - 4:15 

• 

Training your Assessment Administrator 

Your AA 


• 

Review Training Script 

Session #20: Security and 

4:15 - 4:45 

• 

Westat Video 

Privacy 




Session #21: Meet with FM 

4:45 - 5:30 

• 

Assignments 



• 

Report Calls 


2/28/2011 
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Exhibit E-2. Preassessment call log/ROC 


TIMSS&PIRLS Spring 2011 
Preassessment Call Log/ROC 

School Name and ID: 

School Coordinator: 

School Coordinator Email: 

School Phone Number: 

Date of Assessment: 

Completed Forms Missing: (check missing form, if any) 

Class Listing Form □ 

If TIMSS, Student Teacher Linkage Form(s) □ 

If PIRLS, Student Listing Form(s) □ 

IDENTIFY YOURSELF AND ASK TO SPEAK WITH THE SCHOOL COORDINATOR. 
IF HE/SHE IS NOT AVAILABLE, SCHEDULE A TIME TO CALL BACK. 

! 

CALLBACK DATE/TIME 

1. Confirm assessment date. 


TIMSS or PIRLS 
(Circle one or both) 


2. Confirm and/or set time and location. 

DISCUSS TIMING OF SESSIONS AND REMEMBER TIMSS, PIRLS, AND TIMSS-NAEP 
LINK SESSIONS CANNOT BE ADMINISTERED IN THE SAME LOCATION BECAUSE OF 
TIMING DIFFERENCES. AT 4™ GRADE YOU CAN ASSESS 2 TIMSS OR 2 PIRLS AT 
THE SAME TIME AND LOCATION. 
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Exhibit E-2. Preassessment eall log/ROC — Continued 


TIMSS&PIRLS Spring 2011 
Preassessment Call Log/ROC 


3. IF WE HAVE NOT RECEIVED THE CLASS LISTING FORM. 

Was the Information Packet received containing the School Coordinator Handbook and Class 
Listing Form? 

□ Yes (ASK THE SCHOOL COORDINATOR TO SEND THE FORM BACK AND 
SCHEDULE A CALL BACK DATE AND TIME) 

□ No (IF NO, LET THE SCHOOL COORDINATOR KNOW THAT A PACKET WILL BE RE- 
MAILED AND SCHEDULE A CALL BACK DATE AND TIME) 

! 

CALLBACK DATE/TIME 


SEND EMAIL TO TIMSS-PIRLS@WESTAT.COM NOTIFYING THE HOME OFFICE THAT NO 
CLASS LISTING PACKAGE WAS RECEIVED AND THAT THE HOME OFFICE SHOULD 
DIRECTLY CONTACT THE SCHOOL COORDINATOR FOR THE CLASS LISTING 
INFORMATION. 


4. IF WE HAVE NOT RECEIVED THE STUDENT TEACHER LINKAGE FORM/STUDENT 
LISTING FORM: 

Was the packet received containing the Student Teacher Linkage Form(s)/Student Listing 
Form(s)? 

□ Yes (ASK THE SCHOOL COORDINATOR TO LOCATE THE FORM BACK AND 
CONTINUE) 

In the packet are the [Student Teacher Linkage/Student Listing] Forms. There is one form for 
each selected class. On each form we need you to list the students in the sampled class and 
each student’s gender and date of birth. If any of the sampled students have special needs and 
should be excluded, please put the appropriate exclusion code in the Exclusion Status column. 

IF THIS IS A TIMSS SCHOOL: 

We also need you to list each sampled student’s math and science teachers in the appropriate 
columns. 

Do you have any questions about this form? 

ANSWER ANY QUESTIONS. IF YOU DO NOT KNOW THE ANSWER TO A QUESTION TELL 
THE SCHOOL COORDINATOR THAT YOU WILL GET BACK TO HIM/HER. 
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Exhibit E-2. Preassessment eall log/ROC — Continued 


TIMSS&PIRLS Spring 2011 
Preassessment Call Log/ROC 

□ No (IF NO, LET THE SCHOOL COORDINATOR KNOW THAT A PACKET WILL BE 
RE-MAILED AND CONTINUE WITH THE CALL) 


5. Review the Student Tracking Form (STF) to confirm demographic data for each sampled 
student, as well as to confirm any student exclusion or new enrollee information. Update the 
form as necessary. 

IF STUDENT IDS (RATHER THAN STUDENT NAMES) WERE USED ON THE STUDENT 
TRACKING FORM, REMIND THE SC THAT THEIR COPY OF THE STUDENT TRACKING 
FORM (WHICH INCLUDES STUDENT NAMES) SHOULD BE READILY AVAILABLE WHEN 
YOU ARRIVE AT THE SCHOOL. 


6. Review the school and teacher questionnaire status. Record the status on the Teacher 
Tracking Form. 

USE THE SCHOOL ADMINISTRATOR/TEACHER QUESTIONNAIRE STATUS REPORT 
FROM THE FMS TO DETERMINE IF THERE ARE QUESTIONNAIRES THAT ARE NOT 
COMPLETED. IF THERE ARE MISSING QUESTIONNAIRES, TELL THE SC THAT WE WILL 
SEND PAPER VERSIONS TO BE COMPLETED. EMAIL THE HOME OFFICE AT 
TIMSSWH@WESTAT.COM REQUESTING THE NUMBER AND TYPE OF QUESTIONNAIRES 
THAT SHOULD BE SENT. WHEN REQUESTING HARD COPY, PLEASE PROVIDE THE 
FOLLOWING: 

FOR THE SCHOOL QUESTIONNAIRE: 

• SCHOOL ID 

• SCHOOL NAME 

• SCHOOL COORDINATOR NAME 

FOR TEACHER QUESTIONNAIRE: 

• SCHOOL ID 

• SCHOOL NAME 

• SCHOOL COORDINATOR NAME 

• TEACHER ID 

• TEACHER NAME 

• CLASS ID 

• CLASS NAME 

• SUBJECT 
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Exhibit E-2. Preassessment call log/ROC — Continued 


TIMSS&PIRLS Spring 2011 
Preassessment Call Log/ROC 

7. Discuss procedures for gathering the students on assessment day. 


8. (TIMSS 8) Does the school have a policy about calculator use? Calculators are allowed in 
TIMSS but if it is against school policy calculators will not be distributed. 


9. Review parent notification procedures and receipt of consent forms if applicable. If consent is 
needed in the school, discuss with the SC what can be done to obtain consent. 

YOU MUST COLLECT A DATED COPY OF THE LETTER SENT TO THE PARENTS ON 
ASSESSMENT DAY. 


10. Make arrangements for the SC, school principal, or classroom teacher to be available in 
each session to introduce you to the class. 


11. Review the handling of emergencies, problem situations, bathroom breaks and procedures 
for dismissing students at the end of the session. 


U.S. TIMSS and PIRLS 2011 Technical Report and User’s Guide 


E-7 





Exhibit E-2. Preassessment call log/ROC — Continued 


TIMSS&PIRLS Spring 2011 
Preassessment Call Log/ROC 

12. Ask for directions to the school, if you need them. 



13. Ask where you should report when you arrive at the school and tell the School Coordinator 
that you’ll be arriving at least one hour before the scheduled assessment time to get your 
materials ready. 



14. Who should the school honorarium check be made out to? 
CONFIRM SPELLING OF THE NAME (SCHOOL, PRINCIPAL, ETC.). 



15. For school coordinator honorarium, confirm the spelling of the name. 



16. THANK THE SCHOOL COORDINATOR. 
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Exhibit E-2. Preassessment call log/ROC — Continued 


TIMSS&PIRLS Spring 2011 
Preassessment Call Log/ROC 

RECORD EACH CALL 


Record of Contacts: 


Person you spoke to 

Date 

Time 

Outcome 
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Exhibit E-3. PIRLS session script 


PIRLS 




TIMSS & PIRLS 

International Study Center 

Lynch School of Education, Boston College 
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Exhibit E-3. FIRES session script — Continued 


Administering the PIRLS Assessment 

The instructions marked with the CQ symbol and printed in bold font style in the PIRLS 
administration script must be read aloud to the students word for word to ensure that the testing 
sessions are conducted in the same way in all countries. Although you should become familiar 
with these instructions before the actual testing, do not attempt to memorize them. Read these 
instructions exactly as they are written. Comments that are not in bold are not to be read aloud. 
They are instructions for you only. 

To begin the testing session: 

/~~7Make sure that the School Coordinator has recorded the Class ID at the top of the 
PIRLS Test Administration Form. If missing, you can find it on the Student Tracking 
Form, Column [b]. 

/~~7Make sure the students are seated quietly, with nothing on the desk except for a 
pencil. 

/ /Record the current time in Cell (9a) of the PIRLS Test Administration Form. 
/~~7Begin reading the PIRLS Test Administration Script. 
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Exhibit E-3. FIRES session script — Continued 


The PIRLS Session Script 


CQ This school has been chosen to take part in an important international project to 
study how well children around the world can read. Different countries from all 
over the world are taking part in this study. While I talk to you about today’s test, I 
would like you all to be quiet, stay at your desks, and listen carefully, 

CQ Now I will pass out the test booklets. Do not remove the booklet from the envelope 
until I tell you to do so. Each of you will receive one test booklet. Not all booklets are 
the same, 

CQ If you still have any schoolbooks or papers on your desk, please put them away. All 
electronic devices, such as cell phones, portable computers, photo or video cameras, 
must be off and stored away for the duration of the test administration. 


As you hand out the test booklets, make sure that each student receives the booklet specially 
prepared for him or her. You can do that by reading the name on each test booklet, and giving it 
to that student. 

Please make sure that all students who receive Booklet R also receive the PIRLS Reader. 

Do not allow students to open the test booklets until you tell them to. 

Record the student’s participation status in Column 7 (Achievement Session) of the Student 
Tracking Form. If you are administering a makeup session, then use the shaded part of Column 7 
(Achievement Session). 

If a student is absent, put that test booklet aside. Do not give it to anyone else, since each test 
booklet is marked for a specific student. 

If there is a student in the classroom who is not listed on the Student Tracking Form, or an 
originally assigned booklet is damaged, use one of the three spare booklets. 

After the test booklets are passed out and the Student Tracking Form has been completed, say the 
following to the students: 

CQ Please take your test booklet out of the envelope. Turn to the first page in the 

booklet that says “General Directions,” Please read along as I read the Directions 
aloud. 


Please do not admit any students into the testing session at this point. 
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Exhibit E-3. FIRES session script — Continued 


General Directions 

CQ In this test, you will read stories or articles and answer questions about what you 
have read. You may find some parts easy and you may find some parts difficult, 

CQ You will be asked to answer different types of questions. Some of the questions will 
be followed by four choices. You will choose the best answer and fill in the oval next 
to that answer. Example 1 shows this type of question, 

Exumpio 1 

E How many days are there in a week? 

2 days 
4 days 
• 7 days 
10 days 

CQ The oval next to “7 days” is filled in because there are 7 days in a week, 

CQ If you are not sure about the answer to a question, fill in the oval next to the answer 

you think is best, and move on to the next question, 

CQ If you decide to change an answer to a question, completely erase your first choice, 
and then fill in the oval next to your new choice, 

CQ As you work through the questions, you will need to look back at the passages you 
have read to help you answer the questions, 

CQ For some questions you will be asked to write your answers in the space provided in 
your booklet. Example 2 shows one question like this. 


Example 2 

2. WTiere does the little boy go after he finds the book? 



CQ Example 2 has a pencil with a 1 next to it. This means the question is worth 1 point. 
For questions worth 1 point, you need to write a few words, or a sentence, 

CQ Example 3 shows a question with a pencil with a 3 next to it. This means the 

question is worth 3 points. For questions worth 2 or 3 points, you need to explain 
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Exhibit E-3. FIRES session script — Continued 

your view using what you have read in the story or article. You may write in full 
sentences if you wish. 


Example 3 

3. WTiat makes the ending of the story both happy and sad? Use what 
you have read in the stoiy* to help you explain. 



Q You will have 40 minutes to work in your test booklet and then we will take a short 
break. Then, you will work for another 40 minutes, 

CQ Do your best to answer all the questions. If you cannot answer a question, move on 
to the next one. 


Instructions for Part One 

Q At the end of the first part of your booklet, you will see a STOP message. Do not 

continue with the rest of the booklet until you are told to do so. You can review what 
you just worked on until the time is up [or take a book you have with you and read 
quietly at your desk], [I have some activities sheets available for you. If you are 
finished early, please raise your hand to receive one,] 

CQ Do you have any questions? 

When all problems are resolved and you have the students' attention again, record the current 
time in Cell (9b) of the PIRLS Test Administration Form. Then say: 

CQ Turn the page and begin reading the first story in your booklet. 

Begin timing the 40 minutes for Part 1. Record the current time in Cell (10a) of the PIRLS Test 
Administration Form. 

Make sure all students begin working on the correct part of the booklet. Students assigned the 
PIRLS Reader and Booklet R should be reading the first story in the PIRLS Reader. 

Please make sure that students with Booklets 5, 6, and 10 pull out the leaflet that accompanies 
their booklet and use it to answer the questions in the first section of their test booklet. All other 
students should begin reading the story/article in the first section of the test booklet. 
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Exhibit E-3. FIRES session script — Continued 

Remember that you are not allowed to help the students with the test. While the students are 
working, you should move around the room to see that students are working on the eorreet 
seetion of their booklets. 

After 35 minutes have passed, say the following; 

CQ You have 5 minutes left to work on this part of your hooklet. Make sure you try to 
finish answering all of the questions in the first part of your hooklet before the 
break. 

After the last 5 minutes have passed, say: 

C3 Your time is up. Please close your booklets, and put your pencils down. Do not write 
anything more. We will now take a [insert amount of time (up to 30 minutes) that is 
adequate for your students] break, 

Reeord the current time in Cell (10b) of the PIRLS Test Administration Form. 

Break 

If the room will be left unattended during the break, collect the booklets from the students one by 
one. Keep the booklets secure during the break time. You will then redistribute the booklets after 
the break, just like you did at the beginning of the testing session, making sure each student 
receives the same test booklet he/she was working on during the first half of the testing session. 

Part Two 

Record the current time in Cell (11a) of the PIRLS Test Administration Form. 

Make sure all the students are seated. When the students are seated and quiet redistribute the test 
booklets, if necessary. Then, say the following; 

CQ Does everybody have his or her test booklet? 

When all problems are resolved and you have the students' attention again, record the current 
time in Cell (11b) of the PIRLS Test Administration Form and say: 

CQ You are now going to read a story or article and work on questions in the second 
part of your test booklet, 

CQ At the end of the second part of your booklet, you will see a STOP message. If you 
are finished early, you can review your work on the second part of your booklet, 

CQ Open to the second story in your test booklet and you may begin. 

Begin timing the 40 minutes for the second part of the session. Record the current time in Cell 
(12a) of the PIRLS Test Administration Form. 

Make sure all students begin working on the correct part of the booklet. Students assigned the 
PIRLS Reader and Booklet R should be reading the second story in the PIRLS Reader. 

Please make sure that students with Booklets 5, 6, and 10 pull out the leaflet that accompanies 
their booklet and use it to answer the questions in the second section of their test booklet. 
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Exhibit E-3. FIRES session script — Continued 

All other students should begin reading the story/article in the second section of the test booklet. 
Remember that you are not allowed to help the students with the test. While the students are 
working, you should move around the room to see that students are working on the correct 
section of their booklets. If a student is finished early, make sure that he or she has a book to 
read. 

After 35 minutes have passed, say the following: 

CQ You have 5 minutes left to work on this part of your booklet. Make sure you try to 
finish answering all of the questions in the second part of your hooklet. 

After the last 5 minutes have passed, say: 

C3 Your time is up, please stop working. 

Record the current time in Cell (12b) of the PIRLS Test Administration Form and say: 

C3 Now please turn to the page of your hooklet with the pictures of smiley faces and 

answer the questions about how much you liked the things you read. For example, if 
you liked the story a lot, then fill in the oval next to the smiling face and the words 
“I liked it a lot.” 

Make sure that all students are answering the questions on the page of the booklet with the 
smiley faces only. If they need help answering these questions, you are free to help them. Once 
they are finished, please say: 

C3 Thank you for your work. I’m now going to collect all of your extra materials. You 
will need to keep your pencil and test booklet. If you were given a brochure or 
leaflet, please place them in the upper right corner of your desk and I will come 
around and collect them. 

Collect the PIRLS Reader and Day of Hiking leaflet. Make sure you account for all the items that 
you distributed. 

If the Student Questionnaire is going to be administered now, say: 

CQ We will now take a short break. Afterwards, I will ask you to answer a short 
questionnaire. (Please be back on time.) 

If the room will be left unattended during the break, collect the booklets in the envelopes from 
the students one by one. Keep the booklets secure during the break time. You will then 
redistribute the booklets and envelopes after the break, just like you did at the beginning of the 
testing session, making sure each student receives the same test booklet he/she was working on 
during the testing session. 

If the Student Questionnaire is going to be administered at a later time, say: 

C3 Please insert the booklet into the envelope it was in when you received it. 
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Exhibit E-3. FIRES session script — Continued 


Collect all test booklets and keep them secure. Check against the Student Tracking Form to make 
sure that you have received all test booklets. Use the sequential number you assigned to the front 
of the envelope to help in this process. 


Administering the Student Questionnaire 

Before you begin the questionnaire administration: 

/~~7Make sure that you have the corresponding Student Tracking Form and PIRLS Test 
Administration Form. 

/~~^ake sure all the students are in the class; all are seated quietly, and have a pencil. 


When ready, say: 

CQ Now you will complete a short questionnaire. Turn to the colored divider page in 
your hooklet. Do not begin the questionnaire until I tell you to do so, 

CQ The directions are printed at the beginning of your questionnaire. I will also read 

them to you. It is important that you follow the directions very carefully so that you 
understand how to mark your answers. Now open the questionnaire and turn to the 
first page titled “Directions,” 

Make sure that the students have their questionnaires open to the Directions page before 

proceeding. 

CQ Please follow the directions in your questionnaire as I read them aloud. 


Directions 

CQ In this booklet, you will find questions about you and what you think. For each 
question, you should choose the answer you think is best, 

CQ Let us take a few minutes to practice the kinds of questions you will answer in this 
booklet, 

CQ Example 1 is one kind of question you will find in this booklet. 
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Exhibit E-3. FIRES session script — Continued 


Elxample 1 . 

Do you to school? 


Fill in on* oixil only. 
Yes — O 
No - O 


Make sure that all students are following along and are looking at Example 1 in their 
questionnaires. 


CQ In Example 1, the question asks, “Do you go to school?” Below this question are a 
“Yes” and a “No,” Since you all go to school, you should all fill in the oval next to 
“Yes.” 

Give students time to fill in the oval next to “Yes” and make sure they understand how to do it. 
Onee everyone has completed the example, move on to Example 2. 

Make sure that all students are following along and are looking at Example 2 in their 
questionnaires. 

CQ Example 2 is another kind of question you will find in this hooklet. 


Example 2 

How' often do you do these things? 


Fill in one oval for each line. 


Ewry djo- 

Once or 

Once or 

N>vw i 

or almost 

twice a 

twice a 

almost 

every day 

week 

month 

never 


♦ 

i 

♦ 


a) 1 talk with my friends CD <2> <2> 

b) I play sports CD CD CD ® 

c) I ride a skateboard CD CD <3> <D 


CQ This question asks “How often do you do these things?” Letter (a) says, “I talk with 
my friends,” You are given four choices for how often you do this: Every day or 
almost every day; Once or twice a week; Once or twice a month; and Never or 
almost never. 
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Exhibit E-3. FIRES session script — Continued 

CQ Fill in the oval below your answer. For example, if you talk to your friends every 

day or almost every day, fill in the first oval under “Every day or almost every day,” 

Give students time to fill in their answers to all parts of the Example 2 question and make sure 
they understand how to answer this kind of question. Onee everyone has eompleted the example, 
move on to Example 3. 

Make sure that all students are following along and are looking at Example 3 in their 
questionnaires. 

CQ Example 3 is another kind of question you will find in this booklet, 

Example 3 

\Miat do you think? Tt*ll how much you with these 

statements. 


Fill in one oi'al for each line. 




Actm 


Dtfticrre 

DUaCTM 



■ lot 

> litU<> 

a litU* 

a lot 

aJ 

Watching mo\ies is fiin 

CD 

CD 

CD 

CD 

hi 


CD 

Q> 

<D 

CD 

C) 

I do not like waking up early — 

CD 

<2> 

CD 

CD 

di 

I enjoy doing chores 

CD 

O 

<D 

CD 


CQ Example 3 says, “What do you think? Tell how much you agree with these 

statements,” Letter (a) says, “Watching movies is fun,” You are given four choices 
for how much you agree with the statement: Agree a lot. Agree a little. Disagree a 
little, or Disagree a lot, 

CQ Fill in the oval below your answer. For example, if you really agree a lot with that, 
fill in the first oval under “Agree a lot,” If you really disagree a lot, fill in the oval 
under “Disagree a lot,” 

Give students time to fill in their answers to all parts of the Example 3 question and make sure 

they understand how to answer this kind of question. Then eontinue reading the final directions: 

CQ Read each question carefully, and pick the answer you think is best, 

CQ Fill in the oval next to or under your answer, 

CQ If you decide to change your answer, completely erase your first choice. Then, fill in 
the oval next to or under your new answer, 

CQ Ask for help if you do not understand something or are not sure how to answer. 
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Exhibit E-3. FIRES session script — Continued 

CQ Are there any questions before we start? 

If there are questions try to answer them the best you can. If there are no more questions then 
record the current time in Cell (13a) of the PIRLS Test Administration Form and proceed with 
the administration of the questionnaire. 


CQ Turn the page to the first question and begin answering this questionnaire. You will 
have 30 minutes to answer these questions. 

After 30 minutes are up, say: 

CQ Please raise your hand if you have finished answering the questions. 

If not all of the students raise their hands, allow for additional time and say: 

CQ You will have more time to continue answering this questionnaire. If you have 

already finished all the questions, then you can use this time to review your answers. 
Once you have finished, please close your booklet and read quietly at your desk. 

Once all students have finished and have closed their booklets record the current time in Cell 

(13b) of the Test Administration Form. Then say: 

CQ Thank you very much for participating in this study. Your work will help us to 
learn more about our students and schools, 

CQ Please close your booklets and stay seated, 

CQ Please insert the booklet into the envelope it was in when you received it and seal the 
envelope closed. Once you have done this, remove the label that contains your name 
from the envelope. Peel it off and I will collect it. 

Collect all test booklets and envelopes and keep them secure. Check against the Student 

Tracking Form to make sure that you have received all of them. Use the sequential number you 

assigned to the front of the envelope to help in this process. 


Thank you again for your help in conducting this important 

international study. 
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Exhibit E-4. TIMSS fourth-grade session script 



TIMSS 4 
Session Script 



TIMSS fr PIRLS 

International Study Center 

Lynch School of Education, Boston College 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 


Administering the Assessment 

The instructions marked with the CQ symbol and printed in Bold font style in the administration 
script must be read aloud to the students word for word (unless otherwise noted) to ensure that 
the testing sessions are conducted in the same way in all countries. Although you should become 
familiar with these instructions before the actual testing, do not attempt to memorize them. Read 
these instructions exactly as they are written. Comments that are not in bold are not to be read 
aloud. They are instructions for you only. 

To begin the testing session: 

/~~^ake sure that the Class ID is recorded at the top of the Test Administration Form. If 
missing, you can find it on the Student Tracking Form, Column [b]. 

/""/Make sure that students are seated quietly, with nothing on the desk except for a 
pencil. 

/ /Record the current time in Cell (9a) of the Test Administration Form. 

/~~7Begin reading the Administration Script. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 


The Fourth-grade Session Script 

CQ This school has been chosen to take part in an important international project to 

study what students around the world know and can do in mathematics and science. 
Different countries from all over the world are taking part in this study. You will he 
taking a mathematics and science test. While I talk to you about today’s test, I 
would like you all to be quiet, stay at your desks, and listen carefully, 

CQ Now I will pass out the test booklets. Do not remove the booklet from the envelope 
until I tell you to do so. Each of you will receive one test booklet. Not all booklets are 
the same, 

CQ If you still have any schoolbooks, papers, or other materials — for example, a ruler — 
on your desk, please put them away. All electronic devices, such as calculators, cell 
phones, portable computers, photo or video cameras, must be off and stored away 
for the duration of the test administration. 

Students should have only a peneil and his or her booklet for the duration of test administration. 

As you hand out the test booklets, make sure that eaeh student reeeives the booklet speeially 
prepared for him or her. You ean do that by reading the name on eaeh test booklet, and giving it 
to that student. Do not allow students to open the test booklets until you tell them to. 

Reeord the student’s partieipation status in Column 7 (Aehievement Session) of the Student 
Tracking Form. 

If you are administering a makeup session, then use the shaded part of Column 7 (Achievement 
Session). 

If a student is absent, put that test booklet aside. Do not give it to anyone else, since each test 
booklet is marked for a specific student. 

If there is a student in the classroom who is not listed on the Student Tracking Form, or an 
originally assigned booklet is damaged, use one of the three spare booklets, as explained in 
Section 1.2.1. 

After the test booklets are passed out and the Student Tracking Form has been completed, say the 
following to the students: 

CQ Does everybody have his or her test booklet? 

If yes, then continue. If not, find out why and proceed as described before. 

CQ Please take your test booklet out of the envelope. Turn to the first page in the 

booklet that says “General Directions,” Please read along as I read the Directions 
aloud. 

Please do not admit any students into the testing session at this point. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 

General Directions 


Q In this test, you will answer questions in mathematics and science. You may find 
some questions easy, and you may find some questions difficult. Try to answer all 
questions, the difficult ones as well as the easy ones, 

CQ For some questions, you choose the answer you think is correct, and fill in the oval 
next to it. Example 1 shows this kind of question with the circle next to the correct 
answer filled in. 



How many minutes are there in 1 hour? 


<s> 

12 

CD 

24 

• 

60 

<® 

120 


CQ The oval with the letter “C” has been filled in because there are 60 minutes in one 
hour. If you are not sure about the answer to a question, fill in the oval next to the 
answer you think is best, and move on to the next question. 

If you decide to change an answer to a question, completely erase your first choice, 
and then fill in the oval next to your new choice, 

CQ For some questions, you must write your answer in the space below the question. 
Words, drawings, or numbers are used in answers to these questions. Example 3 
shows a question like this. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 


There is a flag in Johns garden. Sometimes it hangs from the flag pole and 
sometimes it waves as shown below. 



What makes the flag wave? 


~TKe lojind 'VVxe. 


m 


m 

CQ Sometimes, to get full points, you will need to explain your answers for science 
questions, or show your work for mathematics questions. Keep your writing and 
calculations as neat as possible, and give mathematics answers in their simplest 
form. For problems in the test involving money, suppose you are in a country where 
the “Zed” is used as the unit of money, like a dollar, 

CQ When you are asked to write an answer, be sure that your handwriting is clear. 
Think carefully about each question, and answer as completely as possible. If you 
are not sure about the answer, give the answer you think is best, and move on to the 
next question, 

CQ You will have 36 minutes to work in the first part of your test booklet. You will then 
have a short break. After the break, you will work for another 36 minutes in the 
second part, 

CQ You may not use a calculator during the test. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 


Instructions for Part 1 

CQ Read each question carefully and answer it as well as you can. If you are not sure 

about the answer to a question, choose or write the answer you think is best, and 
continue with the next question on the test, 

CQ You will have 36 minutes to complete this part of the test, 

CQ Do not start working until you are told to do so, 

CQ At the end of the first part of your booklet, you will see a STOP message. Do not 

continue with the rest of the booklet until you are told to do so. If you have 
completed Part 1 before the time is up, you can review your answers until the time is 
up or take a book you have with you and read quietly at your desk. Do you have 
any questions? 

When all problems are resolved and you have the students' attention again, reeord the eurrent 
time in Cell (9b) of the Test Administration Form. Then say: 

CQ Turn the page to the beginning of Part 1, and start working immediately. 
Remember, you have 36 minutes to complete the first part of the booklet. 

Begin timing the 36 minutes for Part 1. Reeord the eurrent time in Cell (10a) of the Test 
Administration Form. 

Make sure all students begin working on the eorreet part of the booklet. Remember that you are 
not allowed to help the students with the test. While the students are working, you should move 
around the room to see that students are working on the eorreet section of their booklets. 

10 minutes before the testing session ends, say the following: 

CQ You have 10 minutes left before the break. Make sure you try to finish answering all 
of the questions in the first part of your booklet before the break. 

After the last 10 minutes have passed, say: 

CQ Your time is up. Please close your booklets, and put your pencils down. Do not write 
anything more. We will now take a short break. 

Record the current time in Cell (10b) of the Test Administration Form. 

Break 

If the room will be left unattended during the break, collect the booklets from the students one by 
one. Keep the booklets secure during the break time. You will then redistribute the booklets after 
the break, just like you did at the beginning of the testing session, making sure each student 
receives the same test booklet he/she was working on during the first half of the testing session. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 

Instructions for Part 2 

Record the current time in Cell (1 la) of the Test Administration Form. 

Make sure all the students are seated. When the students are seated and quiet redistribute the test 
booklets, if necessary. Then, say the following: 

CQ Does everybody have their test booklet? 

When all problems are resolved and you have the students’ attention again, proceed with the 
instructions for Part 2. 

CQ In Part 2, you will answer more questions in mathematics or science. You will have 
36 minutes to work on this part of the test, 

CQ Read each question carefully and answer it as well as you can. If you are not sure 
about the answer to a question, choose or write the answer you think is best, and 
move on to the next question. 

CQ Do not start working until you are told to do so, 

CQ At the end of the second part of your booklet, you will see a STOP message. If you 
have completed Part 2 before the time is up, you can review your work on the 
second part of your booklet or take a book you have with you and read quietly at 
your desk. Do you have any questions? 

When all problems are resolved and you have the students' attention again, record the current 
time in Cell (11b) of the Test Administration Form and say: 

CQ Turn the page to the beginning of Part 2, and start working immediately. You have 
36 minutes to complete the second part of the booklet. 

Begin timing the 36 minutes for the second part of the session. Record the current time in Cell 
(12a) of the Test Administration Form. 

Remember that you are not allowed to help the students with the test. While the students are 
working, you should move around the room to see that students are working on the correct 
section of their booklets. If a student is finished early, make sure that he or she has a book to 
read. 

10 minutes before the testing session ends, say the following: 

CQ You have 10 minutes left to work on the second part of the booklet. Make sure you 
try to finish answering all of the questions in the second part of your booklet. 

After the last 10 minutes have passed, say: 

HI Your time is up. Please close your booklets, and put your pencils down. Do not write 
anything more. 

Record the current time in Cell (12b) of the Test Administration Form, then say: 

CQ Thank you for your work. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 

If the Student Questionnaire is going to be administered now, say: 

CQ We will now take a short break. Afterwards, I will ask you to answer a short 
questionnaire, (Please be back on time,) 

If the room will be left unattended during the break, collect the booklets in the envelopes from 
the students one by one. Keep the booklets secure during the break time. You will then 
redistribute the booklets and envelopes after the break, just like you did at the beginning of the 
testing session, making sure each student receives the same test booklet he/she was working on 
during the testing session. 

If the Student Questionnaire is going to be administered at a later time, say: 

CQ Please insert the booklet into the envelope it was in when you received it. 

Collect all test booklets and keep them secure. Check against the Student Tracking Form to make 
sure that you have received all test booklets. Use the sequential number you assigned to the front 
of the envelope to help in this process. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 


Administering the Student Questionnaire 

Before you begin the questionnaire administration; 

/ /Make sure that you have the corresponding Student Tracking Form and TIMSS Test 
Administration Form 

/~~7Make sure all the students are in the class are all seated quietly and have a pencil. 


When ready, say: 

CQ Now you will complete a short questionnaire. Turn to the colored divider page in 
your booklet. Do not begin the questionnaire until I tell you to do so. 

CQ The directions are printed at the beginning of your questionnaire. I will also read 

them to you. It is important that you follow the directions very carefully so that you 
understand how to mark your answers. Now open the questionnaire and turn to the 
first page titled “Directions.” 

Make sure that the students have their questionnaires open to the Directions page before 

proceeding. 

CQ Please follow the directions in your questionnaire as I read them aloud. 


Directions 

CQ In this booklet, you will find questions about you and what you think. For each 
question, you should choose the answer you think is best. 

CQ Let us take a few minutes to practice the kinds of questions you will answer in this 
booklet. 

CQ Example 1 is one kind of question you will find in this booklet. 

Make sure that all students are following along and are looking at Example 1 in their 
questionnaires. 

Example 1 

Do you go to school? 


Fill in one oixil only. 
Yes — CD 
No - ® 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 

Q In Example 1, the question asks, “Do you go to school?” Below this question are a 
“Yes” and a “No,” Since you all go to school, you should all fill in the oval next to 
“Yes.” 

Give students time to fill in the circle next to “Yes” and make sure they understand how to do it. 
Once everyone has completed the example, move on to Example 2. 

Make sure that all students are following along and are looking at Example 2 in their 
questionnaires. 

Q Example 2 is another kind of question you will find in this hooklet, 

Example 2 

How' often do you do these things? 

Fill in one oval for each line. 



Every da>' 

Once or 

Once or 

Never or 


or almost 

twice a 

twice a 

almost 


every day 

week 

month 

never 

n \ T tnllr with mv frionH« 

- o 

CD 

<3> 

CD 

hi I nlnv «nnrtc , 

CD 

CD 

CD 

CD 

c; I ride a skateboard 

CD 

CD 

CD 

CD 


Q This question asks “How often do you do these things?” Letter (a) says, “I talk with 
my friends,” You are given four choices for how often you do this: Every day or 
almost every day; Once or twice a week; Once or twice a month; and Never or 
almost never, 

CQ Fill in the oval helow your answer. For example, if you talk to your friends every 
day or almost every day, fill in the first circle under “Every day or almost every 
day,” 

Give students time to fill in their answers to all parts of the Example 2 question and make sure 
they understand how to answer this kind of question. Once everyone has completed the example, 
move on to Example 3. 

Make sure that all students are following along and are looking at Example 3 in their 
questionnaires. 

Q Example 3 is another kind of question you will find in this hooklet. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 


Example 3 

\Sliat do you think? Tell how much you agi'ee ’witli thesie 
sitatement^r 


FilJ in o/j-e OL'al for each lin^. 


a} 

b) 

c) 

d) 


Affree 
a lol 


Agree 
a litUe 


I 


Dianjgree Disagree 
a little a lot 

T T 


Watching monies is fun 

CD 

I like eating ice cream 

- CD 

I dc not like waking tip early 

CD 

I enjoy doing chores — 

CD 


Q> 




<J> 


CS) 




© 


Q> 


© 




CQ Example 3 says, “What do you think? Tell how much you agree with these 

statements,” Letter (a) says, “Watching movies is fun,” You are given four choices 
for how much you agree with the statement: Agree a lot. Agree a little. Disagree a 
little, or Disagree a lot, 

CQ Fill in the oval helow your answer. For example, if you really agree a lot with that, 
fill in the first oval under “Agree a lot,” If you really disagree a lot, fill in the oval 
under “Disagree a lot,” 

Give students time to fill in their answers to all parts of the Example 3 question and make sure 
they understand how to answer this kind of question. Then continue reading the final directions; 

CQ Read each question carefully, and pick the answer you think is best, 

CQ Fill in the oval next to or under your answer, 

CQ If you decide to change your answer, completely erase your first choice. Then, fill in 
the oval next to or under your new answer, 

CQ Ask for help if you do not understand something or are not sure how to answer, 

CQ Are there any questions before we start? 

If there are questions try to answer them the best you can. If there are no more questions then 
record the current time in Cell (13a) of the Test Administration Form and proceed with the 
administration of the questionnaire. 
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Exhibit E-4. TIMSS fourth-grade session seript — Continued 


CQ Turn the page to the first question and begin answering this questionnaire. You will 
have 30 minutes to answer these questions. 

After 30 minutes are up, say: 

CQ Please raise your hand if you have finished answering the questions. 

If not all of the students raise their hands, allow for additional time and say: 

CQ You will have more time to continue answering this questionnaire. If you have 

already finished all the questions, then you can use this time to review your answers. 
Once you have finished, please close your booklet and read quietly at your desk. 

Once all students have finished and have closed their booklets reeord the current time in Cell 

(13b) of the Test Administration Form. Then say: 

CQ Thank you very much for participating in this study. Your work will help us to 
learn more about our students and schools. 

CQ Please close your booklets and stay seated, 

CQ Please insert the booklet into the envelope it was in when you received it and seal the 
envelope closed. Once you have done this, remove the label that contains your name 
from the envelope. Peel it off and I will collect it. 

Collect all test booklets and envelopes and keep them seeure. Cheek against the Student 

Tracking Form to make sure that you have reeeived all of them. Use the sequential number you 

assigned to the front of the envelope to help in this proeess. 


Thank you again for your help in conducting this important 

international study. 
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Exhibit E-5. TIMSS eighth-grade session script 


TIMSS 8 




TIMSS fr PIRLS 

International Study Center 

Lynch School of Education, Boston College 


U.S. TIMSS and PIRLS 2011 Technical Report and User's Guide 


E-33 



Exhibit E-5. TIMSS eighth-grade session script — Continued 


Administering the TIMSS 8 Assessment 

The instructions marked with the CQ symbol and printed in Bold font style in the administration 
script must be read aloud to the students word for word (unless otherwise noted) to ensure that 
the testing sessions are conducted in the same way in all countries. Although you should become 
familiar with these instructions before the actual testing, do not attempt to memorize them. Read 
these instructions exactly as they are written. Comments that are not in bold are not to be read 
aloud. They are instructions for you only. 

To begin the testing session: 

/~~^ake sure that the Class ID is recorded at the top of the Test Administration Form. If 
missing, you can find it on the Student Tracking Form, Column [b]. 

/""/Make sure that students are seated quietly, with nothing on the desk except for a 
pencil and calculator. 

/ /Record the current time in Cell (9a) of the Test Administration Form. 

/~~7Begin reading the Administration Script. 
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Exhibit E-5. TIMSS eighth-grade session script — Continued 


The Eighth-grade Session Script 

CQ This school has been chosen to take part in an important international project to 

study what students around the world know and can do in mathematics and science. 
Different countries from all over the world are taking part in this study. You will he 
taking a mathematics and science test. While I talk to you about today’s test, I 
would like you all to be quiet, stay at your desks, and listen carefully, 

CQ Now I will pass out the test booklets. Do not remove the booklet from the envelope 
until I tell you to do so. Each of you will receive one test booklet. Not all booklets are 
the same, 

CQ If you still have any schoolbooks, papers, or other materials — for example, a ruler — 
on your desk, please put them away. All electronic devices, such as cell phones, 
portable computers, photo or video cameras, must be off and stored away for the 
duration of the test administration. 

Students should have only a peneil, a ealeulator, and his or her booklet for the duration of test 
administration. 

As you hand out the test booklets, make sure that eaeh student reeeives the booklet speeially 
prepared for him or her. You ean do that by reading the name on eaeh test booklet, and giving it 
to that student. Do not allow students to open the test booklets until you tell them to. 

Reeord the student’s participation status in Column 7 (Achievement Session) of the Student 
Tracking Form. 

If you are administering a makeup session, then use the shaded part of Column 7 (Achievement 
Session). 

If a student is absent, put that test booklet aside. Do not give it to anyone else, since each test 
booklet is marked for a specific student. 

If there is a student in the classroom who is not listed on the Student Tracking Form, or an 
originally assigned booklet is damaged, assign one of the three spare booklets. 

After the test booklets are passed out and the Student Tracking Form has been completed, say the 
following to the students: 

CQ Does everybody have his or her test booklet? 

If yes, then continue. If not, find out why and proceed as described before. 

CQ Please take your test booklet out of the envelope. Turn to the first page in the 

booklet that says “General Directions,” Please read along as I read the Directions 
aloud. 

Please do not admit any students into the testing session at this point. 
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Exhibit E-5. TIMSS eighth-grade session seript — Continued 

General Directions 


CQ In this test, you will answer questions in mathematics and science. You may find 
some questions easy, and you may find some questions difficult. Try to answer all 
questions, the difficult ones as well as the easy ones, 

CQ For some questions, you choose the answer you think is correct, and fill in the oval 
next to it. Example 1 shows this kind of question with the circle next to the correct 
answer filled in. 



How many minutes are there in 1 hour? 


<s> 

12 

CD 

24 

• 

60 

CD 

120 


CQ The oval with the letter “C” has been filled in because there are 60 minutes in one 
hour. If you are not sure about the answer to a question, fill in the oval next to the 
answer you think is best, and move on to the next question. 

If you decide to change an answer to a question, completely erase your first choice, 
and then fill in the oval next to your new choice, 

CQ For some questions, you must write your answer in the space below the question. 
Words, drawings, or numbers are used in answers to these questions. Example 3 
shows a question like this. 
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There is a flag in Johns garden. Sometimes it hangs from the flag pole and 
sometimes it waves as shown below. 



What makes the flag wave? 


~TKe lojind 'VVxe. 


m 


m 

CQ Sometimes, to get full points, you will need to explain your answers for science 
questions, or show your work for mathematics questions. Keep your writing and 
calculations as neat as possible, and give mathematics answers in their simplest 
form. For problems in the test involving money, suppose you are in a country where 
the “Zed” is used as the unit of money, like a dollar, 

CQ When you are asked to write an answer, be sure that your handwriting is clear. 
Think carefully about each question, and answer as completely as possible. If you 
are not sure about the answer, give the answer you think is best, and move on to the 
next question, 

CQ You will have 45 minutes to work in the first part of your test booklet. You will then 
have a short break. After the break, you will work for another 45 minutes in the 
second part. 
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Exhibit E-5. TIMSS eighth-grade session script — Continued 


Instructions for Part 1 

CQ Read each question carefully and answer it as well as you can. If you are not sure 
about the answer to a question, choose or write the answer you think is best, and 
continue with the next question on the test, 

CQ You will have 45 minutes to complete this part of the test, 

CQ Do not start working until you are told to do so, 

CQ At the end of the first part of your booklet, you will see a STOP message. Do not 

continue with the rest of the booklet until you are told to do so. If you have 
completed Part 1 before the time is up, you can review your answers until the time is 
up or take a book you have with you and read quietly at your desk. Do you have 
any questions? 


When all problems are resolved and you have the students' attention again, reeord the eurrent 
time in Cell (9b) of the Test Administration Form. Then say: 


CQ Turn the page to the beginning of Part 1, and start working immediately. 
Remember, you have 45 minutes to complete the first part of the booklet. 

Begin timing the 45 minutes for Part 1. Reeord the eurrent time in Cell (10a) of the Test 
Administration Form. 

Make sure all students begin working on the eorreet part of the booklet. Remember that you are 
not allowed to help the students with the test. While the students are working, you should move 
around the room to see that students are working on the eorreet seetion of their booklets. 

10 minutes before the testing session ends, say the following: 

CQ You have 10 minutes left before the break. Make sure you try to finish answering all 
of the questions in the first part of your booklet before the break. 

After the last 10 minutes have passed, say: 

C3 Your time is up. Please close your booklets, and put your pencils down. Do not write 
anything more. We will now take a short break, 

Reeord the eurrent time in Cell (10b) of the Test Administration Form. 

Break 

If the room will be left unattended during the break, eolleet the booklets from the students one by 
one. Keep the booklets seeure during the break time. You will then redistribute the booklets after 
the break, just like you did at the beginning of the testing session, making sure eaeh student 
reeeives the same test booklet he/she was working on during the first half of the testing session. 
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Exhibit E-5. TIMSS eighth-grade session script — Continued 

Instructions for Part 2 

Record the current time in Cell (1 la) of the Test Administration Form. 

Make sure all the students are seated. When the students are seated and quiet redistribute the test 
booklets, if necessary. Then, say the following: 

CQ Does everybody have their test booklet? 

When all problems are resolved and you have the students’ attention again, proceed with the 
instructions for Part 2. 

CQ In Part 2, you will answer more questions in mathematics or science. You will have 
45 minutes to work on this part of the test, 

CQ Read each question carefully and answer it as well as you can. If you are not sure 
about the answer to a question, choose or write the answer you think is best, and 
move on to the next question. 

CQ Do not start working until you are told to do so, 

CQ At the end of the second part of your booklet, you will see a STOP message. If you 
have completed Part 2 before the time is up, you can review your work on the 
second part of your booklet or take a book you have with you and read quietly at 
your desk. Do you have any questions? 

When all problems are resolved and you have the students' attention again, record the current 
time in Cell (11b) of the Test Administration Form and say: 

CQ Turn the page to the beginning of Part 2, and start working immediately. You have 
45 minutes to complete the second part of the booklet. 

Begin timing the 45 minutes for the second part of the session. Record the current time in Cell 
(12a) of the Test Administration Form. 

Remember that you are not allowed to help the students with the test. While the students are 
working, you should move around the room to see that students are working on the correct 
section of their booklets. If a student is finished early, make sure that he or she has either a book 
to read or an activities sheet to work on. 

10 minutes before the testing session ends, say the following: 

CQ You have 10 minutes left to work on the second part of the booklet. Make sure you 
try to finish answering all of the questions in the second part of your booklet. 

After the last 10 minutes have passed, say: 

HI Your time is up. Please close your booklets, and put your pencils down. Do not write 
anything more. 

Record the current time in Cell (12b) of the Test Administration Form, then say: 
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CQ Thank you for your work, I’m now going to collect calculators. You will need to 

keep your pencil and test booklet. If I passed out a calculator for you to use during 
the test, please place it on the upper right hand corner of your desk and I will come 
around and collect it. 

Collect the calculators. Make sure you account for all the calculators that you distributed. 

If the Student Questionnaire is going to be administered now, say: 

CQ We will now take a short break. Afterwards, I will ask you to answer a short 
questionnaire, (Please be back on time.) 

If the room will be left unattended during the break, collect the booklets in the envelopes from 
the students one by one. Keep the booklets secure during the break time. You will then 
redistribute the booklets and envelopes after the break, just like you did at the beginning of the 
testing session, making sure each student receives the same test booklet he/she was working on 
during the testing session. 

If the Student Questionnaire is going to be administered at a later time, say: 

CQ Please insert the booklet into the envelope it was in when you received it. 

Collect all test booklets and keep them secure. Check against the Student Tracking Form to make 
sure that you have received all test booklets. Use the sequential number you assigned to the front 
of the envelope to help in this process. Collect all calculators. 
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Exhibit E-5. TIMSS eighth-grade session script — Continued 


Administering the Student Questionnaire 


Before you begin the questionnaire administration; 


/~~^ake sure that you have the corresponding Student Tracking Form and TIMSS Test 
Administration Form. 

/~~^ake sure all the students are in the class are all seated quietly and have a pencil. 
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Exhibit E-5. TIMSS eighth-grade session seript — Continued 


The Eighth -grade Questionnaire Administration Script 


When ready, say: 

CQ Does everybody have his or her booklet? 

If yes, then continue. If not, find out why and proceed as described before. 


CQ Now you will complete a short questionnaire. Turn to the colored divider page in 
your booklet. Do not begin the questionnaire until I tell you to do so, 

CQ The directions are printed at the beginning of your questionnaire. I will also read 

them to you. It is important that you follow the directions very carefully so that you 
understand how to mark your answers. Now open the questionnaire and turn to the 
first page titled “Directions,” 

Make sure that the students have their questionnaires open to the Directions page before 

proceeding. 

CQ Please follow the directions in your questionnaire as I read them aloud. 


Directions 

CQ In this booklet, you will find questions about you and what you think. For each 
question, you should choose the answer you think is best, 

CQ Let us take a few minutes to practice the kinds of questions you will answer in this 
booklet, 

CQ Example 1 is one kind of question you will find in this booklet. 

Make sure that all students are following along and are looking at Example 1 in their 
questionnaires. 

Example 1 

Do y<m go to jichool? 


Fill in one onal only. 

Yqs- O 
No- © 
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Exhibit E-5. TIMSS eighth-grade session seript — Continued 

Q In Example 1, the question asks, “Do you go to school?” Below this question are a 
“Yes” and a “No,” Since you all go to school, you should all fill in the oval next to 
“Yes.” 

Give students time to fill in the circle next to “Yes” and make sure they understand how to do it. 
Once everyone has completed the example, move on to Example 2. 

Make sure that all students are following along and are looking at Example 2 in their 
questionnaires. 

Q Example 2 is another kind of question you will find in this hooklet, 

Example 2 

How often do you do these things? 


Fill in one oi al for each line. 

Everj' Once or Once or Never or 

or almost twice a twice a almost 

every da>’ week month never 

, i i . 

at I talk with my ftiends • <25 <X> ® 

bl 1 play sports CD <X> <3> 

c) I ride a skateboard CD CD • ■ <3> 


Q This question asks “How often do you do these things?” Letter (a) says, “I talk with 
my friends,” You are given four choices for how often you do this: Every day or 
almost every day; Once or twice a week; Once or twice a month; and Never or 
almost never, 

CQ Fill in the oval helow your answer. For example, if you talk to your friends every 
day or almost every day, fill in the first circle under “Every day or almost every 
day,” 

Give students time to fill in their answers to all parts of the Example 2 question and make sure 
they understand how to answer this kind of question. Once everyone has completed the example, 
move on to Example 3. 

Make sure that all students are following along and are looking at Example 3 in their 
questionnaires. 

Q Example 3 is another kind of question you will find in this hooklet. 
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Example 3 

WTiat do you think? Tell how much you agree with thes»e 
statement!*. 


Fill in one oval for each line. 


Agree Agree Disagree Disagree 

a lot a little a little a lot 

. I i . 

ai Watching mo\'ies is fun CD • (2> <D 

b i I like eating ice cream • <2) <3> CD 

c) I do not like waking up early CD CD • CD 

dl I enjoy doing chores CD CD (D • 


CQ Example 3 says, “What do you think? Tell how much you agree with these 

statements,” Letter (a) says, “Watching movies is fun,” You are given four choices 
for how much you agree with the statement: Agree a lot. Agree a little. Disagree a 
little, or Disagree a lot, 

CQ Fill in the oval below your answer. For example, if you really agree a lot with that, 
fill in the first oval under “Agree a lot,” If you really disagree a lot, fill in the oval 
under “Disagree a lot.” 

Give students time to fill in their answers to all parts of the Example 3 question and make sure 
they understand how to answer this kind of question. Then continue reading the final directions: 

Q Read each question carefully, and pick the answer you think is best, 

Q Fill in tbe circle next to or under your answer, 

CQ If you decide to change your answer, completely erase your first cboice. Then, fill in 

tbe oval next to or under your new answer, 

CQ Ask for help if you do not understand something or are not sure how to answer, 

CQ Are there any questions before we start? 

If there are questions try to answer them the best you can. If there are no more questions then 
record the current time in Cell (13a) of the Test Administration Form and proceed with the 
administration of the questionnaire. 

CQ Turn the page to the first question and begin answering this questionnaire. You will 
have 30 minutes to answer these questions. 

After 30 minutes are up, say: 

CQ Please raise your hand if you have finished answering the questions. 
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Exhibit E-5. TIMSS eighth-grade session script — Continued 

If not all of the students raise their hands, allow for additional time and say: 

CQ You will have more time to continue answering this questionnaire. If you have 

already finished all the questions, then you can use this time to review your answers. 
Once you have finished, please close your booklet and read quietly at your desk. 

Once all students have finished and have elosed their booklets reeord the eurrent time in Cell 

(13b) of the Test Administration Form. Then say: 

CQ Thank you very much for participating in this study. Your work will help us to 
learn more about our students and schools. 

CQ Please close your booklets and stay seated. 

CQ Please insert the booklet into the envelope it was in when you received it and seal the 
envelope closed. Once you have done this, remove the label that contains your name 
from the envelope. Peel it off and I will collect it. 

Collect all test booklets and envelopes and keep them seeure. Cheek against the Student 

Traeking Form to make sure that you have reeeived all of them. Use the sequential number you 

assigned to the front of the envelope to help in this proeess. 


Thank you again for your help in conducting this important 

international study. 
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Exhibit E-6. TIMSS-NAEP link session script 


TIMSS-NAEP Link 
Session Script 




TIMSS fr PIRLS 

International Study Center 

Lynch School of Education, Boston College 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 


Administering the TIMSS-NAEP Link Assessment 

The instructions marked with the CQ symbol and printed in Bold font style in the administration 
script must be read aloud to the students word for word (unless otherwise noted) to ensure that 
the testing sessions are conducted in the same way in all countries. Although you should become 
familiar with these instructions before the actual testing, do not attempt to memorize them. Read 
these instructions exactly as they are written. Comments that are not in bold are not to be read 
aloud. They are instructions for you only. 

To begin the testing session: 

/~~^ake sure that the Class ID is recorded at the top of the Test Administration Form. If 
missing, you can find it on the Student Tracking Form, Column [b]. 

/""/Make sure that students are seated quietly, with nothing on the desk except for a 
pencil and calculator. 

/ /Record the current time in Cell (9a) of the Test Administration Form. 

/~~7Begin reading the Administration Script. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 

The TIMSS-NAEP Link Class Session Script 

CQ This school has been chosen to take part in an important international project to 

study what students around the world know and can do in mathematics and science. 
Different countries from all over the world are taking part in this study. You will he 
taking a mathematics and science test. While I talk to you about today’s test, I 
would like you all to be quiet, stay at your desks, and listen carefully. 

CQ Now I will pass out the test booklets. Do not remove the booklet from the envelope 
until I tell you to do so. Each of you will receive one test booklet. Not all booklets are 
the same. 

CQ If you still have any schoolbooks, papers, or other materials — for example, a ruler — 
on your desk, please put them away. All electronic devices, such as cell phones, 
portable computers, photo or video cameras, must be off and stored away for the 
duration of the test administration. 

Students should have only a pencil, calculator, and his or her booklet for the duration of test 
administration. 

As you hand out the test booklets, make sure that each student receives the booklet specially 
prepared for him or her. You can do that by reading the name on each security envelope, and 
giving it to that student. Do not allow students to open the test booklets until you tell them to. 

Record the student’s participation status in Column 7 (Achievement Session) of the Student 
Tracking Form. 

If you are administering a makeup session, then use the shaded part of Column 7 (Achievement 
Session). 

If a student is absent, put that test booklet aside. Do not give it to anyone else, since each test 
booklet is marked for a specific student. 

If there is a student in the classroom who is not listed on the Student Tracking Form, or an 
originally assigned booklet is damaged, use one of the three spare booklets. After the test 
booklets are passed out and the Student Tracking Form has been completed, say the following to 
the students: 

CQ Does everybody have his or her test booklet? 

If yes, then continue. If not, find out why and proceed as described before. 

CQ Please take your test booklet out of the envelope. If your booklet has an “N” in the 
top left corner, please put your calculator away. Your booklet will instruct you 
when you may use your calculator. 

CQ Turn to the first page in the booklet that says “General Directions”, Please read 
along as I read the Directions aloud. 

Please do not admit any students into the testing session at this point. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 

General Directions 


CQ In this test, you will answer questions in mathematics and science. You may find 
some questions easy, and you may find some questions difficult. Try to answer all 
questions, the difficult ones as well as the easy ones, 

CQ For some questions, you choose the answer you think is correct, and fill in the oval 
next to it. Example 1 shows this kind of question with the circle next to the correct 
answer filled in. 



How many minutes are there in 1 hour? 


<s> 

12 

CD 

24 

• 

60 

CD 

120 


CQ The oval with the letter “C” has been filled in because there are 60 minutes in one 
hour. If you are not sure about the answer to a question, fill in the oval next to the 
answer you think is best, and move on to the next question. 

If you decide to change an answer to a question, completely erase your first choice, 
and then fill in the oval next to your new choice, 

CQ For some questions, you must write your answer in the space below the question. 
Words, drawings, or numbers are used in answers to these questions. Example 3 
shows a question like this. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 


There is a flag in Johns garden. Sometimes it hangs from the flag pole and 
sometimes it waves as shown below. 



What makes the flag wave? 


~TKe lojind 'VVxe. 


m 


m 

CQ Sometimes, to get full points, you will need to explain your answers for science 
questions, or show your work for mathematics questions. Keep your writing and 
calculations as neat as possible, and give mathematics answers in their simplest 
form. For some problems in the test involving money, suppose you are in a country 
where the “Zed” is used as the unit of money, like dollars, 

CQ When you are asked to write an answer, be sure that your handwriting is clear. 
Think carefully about each question, and answer as completely as possible. If you 
are not sure about the answer, give the answer you think is best, and move on to the 
next question, 

CQ You will have 47 minutes to work in the first part of your test booklet. You will then 
have a short break. After the break, you will work for another 45 minutes in the 
second part. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 


Instructions for Part 1 

CQ Read each question carefully and answer it as well as you can. If you are not sure 

about the answer to a question, choose or write the answer you think is best, and 
continue with the next question on the test, 

CQ You will have 47 minutes to complete this part of the test, 

CQ Do not start working until you are told to do so, 

CQ At the end of the first part of your booklet, you will see a STOP message. Do not 

continue with the rest of the booklet until you are told to do so. If you have 
completed Part 1 before the time is up, you can review your answers until the time is 
up or take a book you have with you and read quietly at your desk. Do you have 
any questions? 

When all problems are resolved and you have the students' attention again, reeord the eurrent 
time in Cell (9b) of the Test Administration Form. Then say: 

CQ Turn the page to the beginning of Part 1, and start working immediately. 
Remember, you have 47 minutes to complete the first part of the booklet. 

Begin timing the 47 minutes for Part 1. Reeord the eurrent time in Cell (10a) of the Test 
Administration Form. 

Make sure all students begin working on the eorreet part of the booklet. Remember that you are 
not allowed to help the students with the test exeept if they are eonfused about items involving 
eurrency. Refer to your Student Questionnaire Guide for responses to questions about dollars 
and eents and Zeds. While the students are working, you should move around the room to see 
that students are working on the eorreet seetion of their booklets. 

Also, you should be monitoring the students to make sure those students that are restricted from 
using a calculator in the NAEP items are not using one. 

10 minutes before the testing session ends, say the following: 

CQ You have 10 minutes left before the break. Make sure you try to finish answering all 
of the questions in the first part of your booklet before the break. 

After the last 10 minutes have passed, say: 

CQ Your time is up. Please close your booklets, and put your pencils down. Do not write 
anything more. We will now take a short break. 

Record the current time in Cell (10b) of the Test Administration Form. 


U.S. TIMSS and PIRLS 2011 Technical Report and User's Guide 


E-51 



Exhibit E-6. TIMSS-NAEP link session script — Continued 

Break 

If the room will be left unattended during the break, eollect the booklets from the students one by 
one. Keep the booklets secure during the break time. You will then redistribute the booklets after 
the break, just like you did at the beginning of the testing session, making sure each student 
receives the same test booklet he/she was working on during the first half of the testing session. 

Instructions for Part 2 

Record the current time in Cell (1 la) of the Test Administration Form. 

Make sure all the students are seated. When the students are seated and quiet redistribute the test 
booklets, if necessary. Then, say the following: 

CQ Does everybody have their test booklet? 

When all problems are resolved and you have the students’ attention again, proceed with the 
instructions for Part 2. 

CQ In Part 2, you will answer more questions in mathematics or science. You will have 
45 minutes to work on this part of the test, 

CQ Read each question carefully and answer it as well as you can. If you are not sure 
about the answer to a question, choose or write the answer you think is best, and 
move on to the next question, 

CQ Do not start working until you are told to do so, 

CQ At the end of the second part of your booklet, you will see a STOP message. If you 
have completed Part 2 before the time is up, you can review your work on the 
second part of your booklet or take a book you have with you and read quietly at 
your desk. Do you have any questions? 

When all problems are resolved and you have the students' attention again, record the current 
time in Cell (11b) of the Test Administration Form and say: 

CQ Turn the page to the beginning of Part 2, and start working immediately. You have 
45 minutes to complete the second part of the booklet. 

Begin timing the 45 minutes for the second part of the session. Record the current time in Cell 
(12a) of the Test Administration Form. 

Remember that you are not allowed to help the students with the test. While the students are 
working, you should move around the room to see that students are working on the correct 
section of their booklets. If a student is finished early, make sure that he or she has either a book 
to read or an activities sheet to work on. 

10 minutes before the testing session ends, say the following: 

CQ You have 10 minutes left to work on the second part of the booklet. Make sure you 
try to finish answering all of the questions in the second part of your booklet. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 
After the last 10 minutes have passed, say: 

CQ Your time is up. Please close your booklets, and put your pencils down. Do not write 
anything more. 

Record the current time in Cell (12b) of the Test Administration Form, then say: 

CQ Thank you for your work, I’m now going to collect all of your extra materials. You 
will need to keep your pencil and test booklet. If you have a packet of materials, 
make sure to place the materials back in the bag. Please place any calculators or 
extra materials in the upper right corner of your desk and I will come around and 
collect them. 

Collect the calculators and manipulatives. Make sure you account for all items that you 
distributed. 

If the Student Questionnaire is going to be administered now, say: 

CQ We will now take a short break. Afterwards, I will ask you to answer a short 
questionnaire, (Please be back on time,) 

If the room will be left unattended during the break, collect the booklets in the envelopes from 
the students one by one. Keep the booklets secure during the break time. You will then 
redistribute the booklets and envelopes after the break, just like you did at the beginning of the 
testing session, making sure each student receives the same test booklet he/she was working on 
during the testing session. 

If the Student Questionnaire is going to be administered at a later time, say: 

CQ Please insert the booklet into the envelope it was in when you received it. 

Collect all test booklets and keep them secure. Check against the Student Tracking Form to make 
sure that you have received all test booklets. Use the sequential number you assigned to the front 
of the envelope to help in this process. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 


Administering the Student Questionnaire 


Before you begin the questionnaire administration: 


/~~7Make sure that you have the corresponding Student Tracking Form and TIMSS Test 
Administration Form. 

/~~7Make sure all the students are in the class are all seated quietly and have a pencil. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 


The TIMSS-NAEP Link Class Questionnaire Administration Script 


When ready, say: 

CQ Does everybody have his or her booklet? 

If yes, then continue. If not, find out why and proceed as described before. 


CQ Now you will complete a short questionnaire. Turn to the colored divider page in 
your booklet. Do not begin the questionnaire until I tell you to do so, 

CQ The directions are printed at the beginning of your questionnaire. I will also read 

them to you. It is important that you follow the directions very carefully so that you 
understand how to mark your answers. Now open the questionnaire and turn to the 
first page titled “Directions,” 

Make sure that the students have their questionnaires open to the Directions page before 

proceeding. 

CQ Please follow the directions in your questionnaire as I read them aloud. 


Directions 

CQ In this booklet, you will find questions about you and what you think. For each 
question, you should choose the answer you think is best, 

CQ Let us take a few minutes to practice the kinds of questions you will answer in this 
booklet, 

CQ Example 1 is one kind of question you will find in this booklet. 

Make sure that all students are following along and are looking at Example 1 in their 
questionnaires. 

Example 1 

Do you go to iifhool? 


Fill m one anal only. 

Yes- O 
No- © 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 

Q In Example 1, the question asks, “Do you go to school?” Below this question are a 
“Yes” and a “No,” Since you all go to school, you should all fill in the oval next to 
“Yes.” 

Give students time to fill in the circle next to “Yes” and make sure they understand how to do it. 
Once everyone has completed the example, move on to Example 2. 

Make sure that all students are following along and are looking at Example 2 in their 
questionnaires. 

Q Example 2 is another kind of question you will find in this hooklet, 

Example 2 

How often do you do these things? 


Fill in one oi al for each line. 

Everj' Once or Once or Never or 

or almost twice a twice a almost 

every da>’ week month never 

, i i . 

at I talk with my ftiends • <25 <X> ® 

bl 1 play sports CD <X> <3> 

c) I ride a skateboard CD CD • ■ <3> 


Q This question asks “How often do you do these things?” Letter (a) says, “I talk with 
my friends,” You are given four choices for how often you do this: Every day or 
almost every day; Once or twice a week; Once or twice a month; and Never or 
almost never, 

CQ Fill in the oval helow your answer. For example, if you talk to your friends every 
day or almost every day, fill in the first circle under “Every day or almost every 
day,” 

Give students time to fill in their answers to all parts of the Example 2 question and make sure 
they understand how to answer this kind of question. Once everyone has completed the example, 
move on to Example 3. 

Make sure that all students are following along and are looking at Example 3 in their 
questionnaires. 

Q Example 3 is another kind of question you will find in this hooklet. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 


Example 3 

WTial do you think? Tell how much you agree with these 
statements. 


Fill in one oi'al for each line. 


at 

bl 

c) 

di 


Agree Agree Disagree Disagree 

a lot a little a little a lot 

▼ T 

Watching mo\ies is fun <D • <X> <X> 

1 like eating ice cream • « <D (X> <D 

1 do not like waking up early CD (2> • CD 

I enjoy doing chores CD CD d> • 


CQ Example 3 says, “What do you think? Tell how much you agree with these 

statements,” Letter (a) says, “Watching movies is fun,” You are given four choices 
for how much you agree with the statement: Agree a lot. Agree a little. Disagree a 
little, or Disagree a lot, 

CQ Fill in the oval helow your answer. For example, if you really agree a lot with that, 
fill in the first oval under “Agree a lot,” If you really disagree a lot, fill in the oval 
under “Disagree a lot,” 

Give students time to fill in their answers to all parts of the Example 3 question and make sure 
they understand how to answer this kind of question. Then continue reading the final directions; 

CQ Read each question carefully, and pick the answer you think is best, 

CQ Fill in the oval next to or under your answer, 

CQ If you decide to change your answer, completely erase your first choice. Then, fill in 
the oval next to or under your new answer, 

CQ Ask for help if you do not understand something or are not sure how to answer, 

CQ Are there any questions before we start? 

If there are questions try to answer them the best you can. If there are no more questions then 
record the current time in Cell (13a) of the Test Administration Form and proceed with the 
administration of the questionnaire. 
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Exhibit E-6. TIMSS-NAEP link session script — Continued 


CQ Turn the page to the first question and begin answering this questionnaire. You will 
have 30 minutes to answer these questions. 

After 30 minutes are up, say: 

CQ Please raise your hand if you have finished answering the questions. 

If not all of the students raise their hands, allow for additional time and say: 

CQ You will have more time to continue answering this questionnaire. If you have 

already finished all the questions, then you can use this time to review your answers. 
Once you have finished, please close your booklet and read quietly at your desk. 

Once all students have finished and have closed their booklets reeord the current time in Cell 

(13b) of the Test Administration Form. Then say: 

CQ Thank you very much for participating in this study. Your work will help us to 
learn more about our students and schools. 

CQ Please close your booklets and stay seated, 

CQ Please insert the booklet into the envelope it was in when you received it and seal the 
envelope closed. Once you have done this, remove the label that contains your name 
from the envelope. Peel it off and I will collect it. 

Collect all test booklets and envelopes and keep them seeure. Cheek against the Student 

Tracking Form to make sure that you have reeeived all of them. Use the sequential number you 

assigned to the front of the envelope to help in this proeess. 


Thank you again for your help in conducting this important 

international study. 
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